Save Debug Logs Only When Your Python Script Crashes

An in-memory ring buffer of recent debug events, written to disk only on a crash, a hang or a logged error, in under 200 lines of standard-library Python.

·
10 min read
·
AI-assisted

AI tooling disclosure

Generative AI materially contributed to this article.

Human review confirmed by the author.

Save Debug Logs Only When Your Python Script Crashes

Log at INFO and the output stays readable, but the line you need after a failure is almost always a DEBUG line you didn't keep. Log at DEBUG and you keep it, along with a flood of everything else: more disk, more noise to search through, and more chances of writing a secret into a file that lives forever.

Aircraft solve the same problem with a flight recorder: record continuously, keep only the recent past, and read it only after something goes wrong. The idea carries over to a script or a small service almost unchanged. This post builds one as a standard logging handler: no dependencies, no changes to your existing log calls, and under 200 lines.

How it works

The recorder is a ring buffer, a deque with a maximum length, fed by a logging handler that sees every level. When it fills up, the oldest events fall off the end. A note() method adds breadcrumbs for things your log calls don't cover. Nothing touches the disk until a trigger fires:

Trigger

Why it exists

Unhandled exception, in the main thread or any other

The classic crash. Ctrl-C arrives the same way, which is handy for interrupting a hung loop.

A logged ERROR, at most one dump a minute by default

Services usually catch exceptions and log them, so the unhandled-exception hook never fires.

SIGTERM

A supervisor or a timeout is killing the process, and this is its last chance to explain itself.

SIGUSR1

Snapshot a process that looks stuck without killing it.

A dump file holds the reason, the traceback if there is one, the stack of every thread, and the last N events with timestamps relative to the moment of the dump.

flowchart LR
    A["Your code<br/>logging calls and note()"] --> B[("Ring buffer<br/>last N events")]
    A --> C["Console or file handler<br/>INFO and up"]
    T{"Any trigger<br/>fires"} --> D["Dump file"]
    B -.->|read at dump time| D

Why not the standard library's MemoryHandler?

logging.handlers.MemoryHandler looks like the obvious fit, but when its buffer fills up it flushes everything to its target instead of discarding the oldest records. In a quick test with a capacity of 3 and 7 debug records, the target received 6 of them. It is a batching tool, so every record eventually reaches the disk, which is exactly what we're trying to avoid.

The code

Save this as recorder.py. It needs Python 3.8 or newer (for threading.excepthook); I ran everything here on 3.12 on Linux.

import collections
import faulthandler
import logging
import os
import platform
import re
import signal
import sys
import threading
import time
import traceback

# Best-effort scrubbing, applied when a dump is written (never to the live buffer).
# The part wrapped in (?P<keep>...) survives; everything after it is replaced.
DEFAULT_REDACT = (
    r"(?i)(?P<keep>\b(?:authorization|password|passwd|secret|token|api[-_]?key)\b"
    r"[\"']?\s*[:=]\s*[\"']?(?:(?:Bearer|Basic|Digest|Token)\s+)?)[^\s,;&\"']+",
)


class FlightRecorder(logging.Handler):
    def __init__(self, directory="crashdumps", capacity=500, max_dumps=10,
                 max_msg=500, dump_on=logging.ERROR, cooldown=60, redact=DEFAULT_REDACT):
        super().__init__(level=logging.DEBUG)
        self.directory, self.max_dumps, self.max_msg = directory, max_dumps, max_msg
        self.dump_on, self.cooldown = dump_on, cooldown
        self.redact = [re.compile(p) for p in redact]
        self.events = collections.deque(maxlen=capacity)  # oldest events fall off the end
        self._lock = threading.RLock()  # re-entrant: a signal can land while we hold it
        self._last_dump = 0.0
        self._count = 0

    # ---- recording -------------------------------------------------------
    def emit(self, record):
        try:
            try:
                msg = record.getMessage()
            except Exception:
                msg = f"<unformattable message: {record.msg!r}>"
            exc = record.exc_info if record.exc_info and record.exc_info[0] else None
            if exc:
                msg += f"  !! {exc[0].__name__}: {exc[1]}"
            self._add(record.created, record.levelname, record.name, record.threadName, msg)
            if self.dump_on is not None and record.levelno >= self.dump_on:
                self.dump(f"{record.levelname} logged by {record.name}", exc, throttle=True)
        except Exception:
            self.handleError(record)

    def note(self, event, **fields):
        """A breadcrumb: record a step or a value that logging calls don't cover."""
        parts = [event]
        for key, value in fields.items():
            try:
                parts.append(f"{key}={value!r}")
            except Exception:
                parts.append(f"{key}=<unrepresentable>")
        self._add(time.time(), "NOTE", "breadcrumb", threading.current_thread().name, " ".join(parts))

    def _add(self, ts, level, name, thread, msg):
        if len(msg) > self.max_msg:  # store text, not objects: bounded and can't change later
            msg = msg[:self.max_msg] + f"...[{len(msg) - self.max_msg} more chars]"
        with self._lock:
            self.events.append((ts, level, name, thread, msg))

    # ---- dumping ---------------------------------------------------------
    def dump(self, reason, exc_info=None, throttle=False):
        """Write the buffer to a file. Never raises; returns the path or None."""
        try:
            now = time.time()
            with self._lock:
                if throttle and now - self._last_dump < self.cooldown:
                    return None
                self._last_dump = now
                self._count += 1
                n, events = self._count, list(self.events)

            out = [f"reason  : {reason}",
                   f"time    : {time.strftime('%Y-%m-%d %H:%M:%S %z', time.localtime(now))}",
                   f"process : pid {os.getpid()}, argv {sys.argv}",
                   f"python  : {platform.python_version()} on {platform.platform()}"]
            if exc_info:
                out += ["", "== EXCEPTION =="]
                out += "".join(traceback.format_exception(*exc_info)).rstrip("\n").split("\n")
            out += ["", "== THREAD STACKS =="]
            names = {t.ident: t.name for t in threading.enumerate()}
            for ident, frame in sys._current_frames().items():
                out.append(f"-- {names.get(ident, ident)}")
                out += "".join(traceback.format_stack(frame)).rstrip("\n").split("\n")
            out += ["", f"== LAST {len(events)} EVENTS (seconds relative to this dump) =="]
            out += [f"{ts - now:+9.3f}  {lvl:<8} [{thread}] {name}: {msg}"
                    for ts, lvl, name, thread, msg in events]
            text = self._scrub("\n".join(out)) + "\n"

            os.makedirs(self.directory, mode=0o700, exist_ok=True)
            stamp = time.strftime("%Y%m%d-%H%M%S", time.localtime(now))
            path = os.path.join(self.directory, f"dump-{stamp}-{os.getpid()}-{n}.txt")
            fd = os.open(path, os.O_WRONLY | os.O_CREAT | os.O_EXCL, 0o600)  # owner-only
            with os.fdopen(fd, "w", encoding="utf-8") as f:
                f.write(text)
            self._prune()
            print(f"flight recorder: wrote {path}", file=sys.stderr)
            return path
        except Exception as e:
            print(f"flight recorder: dump failed: {e!r}", file=sys.stderr)
            return None

    def _scrub(self, text):
        for rx in self.redact:
            text = rx.sub(lambda m: (m.groupdict().get("keep") or "") + "[REDACTED]", text)
        return text

    def _prune(self):
        dumps = sorted((os.path.join(self.directory, f) for f in os.listdir(self.directory)
                        if f.startswith("dump-")), key=os.path.getmtime)
        for old in dumps[:-self.max_dumps]:
            os.remove(old)

    # ---- wiring ----------------------------------------------------------
    def install(self):
        """Attach to the root logger and hook crashes and signals. Call it once,
        from the main thread, after your own logging and signal setup."""
        root = logging.getLogger()
        previous = root.level or logging.WARNING
        for handler in root.handlers:  # keep existing handlers as quiet as they were
            if handler.level == logging.NOTSET:
                handler.setLevel(previous)
        root.setLevel(logging.DEBUG)  # the recorder now sees everything
        root.addHandler(self)

        prev_hook = sys.excepthook

        def on_crash(etype, value, tb):
            self.dump(f"unhandled {etype.__name__}", (etype, value, tb))
            prev_hook(etype, value, tb)
        sys.excepthook = on_crash

        prev_thread_hook = threading.excepthook

        def on_thread_crash(args):
            if args.exc_type is not SystemExit:
                who = args.thread.name if args.thread else "unknown thread"
                self.dump(f"unhandled {args.exc_type.__name__} in {who}",
                          (args.exc_type, args.exc_value, args.exc_traceback))
            prev_thread_hook(args)
        threading.excepthook = on_thread_crash

        os.makedirs(self.directory, mode=0o700, exist_ok=True)
        self._fault_log = open(os.path.join(self.directory, "faulthandler.log"), "a")
        faulthandler.enable(self._fault_log)  # hard crashes: stacks only, the buffer is lost
        if hasattr(faulthandler, "register") and hasattr(signal, "SIGUSR2"):
            faulthandler.register(signal.SIGUSR2, file=self._fault_log)  # works even if Python is stuck in C

        prev_term = signal.getsignal(signal.SIGTERM)

        def on_term(signum, frame):
            self.dump("SIGTERM received")
            if callable(prev_term):
                prev_term(signum, frame)  # let the app shut down its own way
            elif prev_term != signal.SIG_IGN:
                signal.signal(signum, signal.SIG_DFL)
                os.kill(os.getpid(), signum)  # die with the usual exit status
        signal.signal(signal.SIGTERM, on_term)

        if hasattr(signal, "SIGUSR1"):  # live snapshot of a stuck process, no kill needed
            signal.signal(signal.SIGUSR1, lambda signum, frame: self.dump("SIGUSR1 snapshot"))
        return self

A few decisions are worth explaining:

  • Existing handlers keep their level. Raising the root logger to DEBUG would make a console handler with no level of its own start printing every debug line. install() first pins such handlers to the previous root level. Call it after logging.basicConfig(...), because a handler added later with no level will show debug output.

  • Events are stored as truncated text, not objects. The buffer can't keep large objects alive, and a message can't change after the fact. At the defaults the full buffer held about 330 KB in my test, with every slot at the maximum message length.

  • The lock is an RLock for a reason. Python runs signal handlers on the main thread between bytecodes, which can be while that thread is inside _add() holding the lock. With a plain Lock, the handler waits forever for a lock its own thread holds. I tested this by firing 200 SIGUSR1 signals during heavy logging: the RLock version finished, and the plain-Lock version hung until I killed it.

  • SIGTERM chains instead of replacing. After dumping, it calls whatever handler your app had installed. If there was none, it re-delivers the default signal, so the process still exits as "killed by signal 15" rather than with a made-up exit code.

  • Dumps are treated as sensitive. Files are owner-only (0600) in a 0700 directory, only the newest 10 are kept, and secrets are scrubbed at dump time by a regex list. That list covers password=..., Authorization: Bearer ... and api_key: ... shapes, and it is best effort, so add patterns for whatever your logs contain.

Using it

import logging
from recorder import FlightRecorder

logging.basicConfig(level=logging.INFO)    # your normal setup, unchanged
rec = FlightRecorder(capacity=500).install()

rec.note("batch start", batch_id=batch.id, size=len(batch))   # optional breadcrumbs

Call install() once, from the main thread, because Python only allows signal handlers to be set there.

Crash test

This script imports pages of rows from a pretend feed. Page 4 contains one row with no price, and the code doesn't expect that.

import logging
import time

from recorder import FlightRecorder

logging.basicConfig(level=logging.INFO, format="%(asctime)s %(levelname)s %(message)s")
rec = FlightRecorder(capacity=200).install()
log = logging.getLogger("importer")


def fetch_page(n):
    log.debug("GET /feed?page=%d", n)
    time.sleep(0.001)  # pretend network
    rows = [{"id": n * 100 + i, "price": f"{1000 + i * 37:,}.00", "currency": "USD"}
            for i in range(5)]
    if n == 4:
        rows[3]["price"] = None  # the one bad row in the feed
    log.debug("page %d -> %d rows", n, len(rows))
    return rows


def parse(row):
    log.debug("parsing row %r", row)
    return float(row["price"].replace(",", ""))


total = 0.0
for page in range(1, 8):
    rec.note("page start", page=page, total=round(total, 2))
    for row in fetch_page(page):
        total += parse(row)
    log.info("page %d done, running total %.2f", page, total)

With the console at INFO, this is everything you get (paths shortened):

2026-11-14 08:03:30,534 INFO page 1 done, running total 5370.00
2026-11-14 08:03:30,535 INFO page 2 done, running total 10740.00
2026-11-14 08:03:30,537 INFO page 3 done, running total 16110.00
flight recorder: wrote crashdumps/dump-20261114-080330-96-1.txt
Traceback (most recent call last):
  File "demo_crash.py", line 31, in <module>
    total += parse(row)
             ^^^^^^^^^^
  File "demo_crash.py", line 24, in parse
    return float(row["price"].replace(",", ""))
                 ^^^^^^^^^^^^^^^^^^^^
AttributeError: 'NoneType' object has no attribute 'replace'

The traceback says what failed but not which row, and nothing logged at INFO can tell you. The dump can (excerpt, with the middle trimmed):

reason  : unhandled AttributeError
time    : 2026-11-14 08:03:30 +0000
process : pid 96, argv ['demo_crash.py']

== EXCEPTION ==
[same traceback as above]

== LAST 34 EVENTS (seconds relative to this dump) ==
[...]
   -0.001  NOTE     [MainThread] breadcrumb: page start page=4 total=16110.0
   -0.001  DEBUG    [MainThread] importer: GET /feed?page=4
   -0.000  DEBUG    [MainThread] importer: page 4 -> 5 rows
   -0.000  DEBUG    [MainThread] importer: parsing row {'id': 400, 'price': '1,000.00', 'currency': 'USD'}
   -0.000  DEBUG    [MainThread] importer: parsing row {'id': 401, 'price': '1,037.00', 'currency': 'USD'}
   -0.000  DEBUG    [MainThread] importer: parsing row {'id': 402, 'price': '1,074.00', 'currency': 'USD'}
   -0.000  DEBUG    [MainThread] importer: parsing row {'id': 403, 'price': None, 'currency': 'USD'}

The offending row is the last event: id 403, price of None. The -0.000 offsets are an artifact of this toy script running in milliseconds; in a real service they show how long before the failure each event happened.

Hang test

Crashes are the easy case. The harder one is a process that is still running but doing nothing. This script deadlocks on purpose: the main thread holds lock_b and wants lock_a, while a worker holds lock_a and wants lock_b.

import logging
import threading
import time

from recorder import FlightRecorder

logging.basicConfig(level=logging.INFO, format="%(asctime)s %(levelname)s %(message)s")
FlightRecorder().install()
log = logging.getLogger("worker")

lock_a, lock_b = threading.Lock(), threading.Lock()


def worker():
    with lock_a:
        log.debug("worker holds lock_a, wants lock_b")
        time.sleep(0.2)
        with lock_b:
            log.info("worker done")


threading.Thread(target=worker, name="worker-1", daemon=True).start()
with lock_b:
    log.debug("main holds lock_b, wants lock_a")
    time.sleep(0.2)
    with lock_a:
        log.info("main done")

Run it in the background, ask for a snapshot, then stop it:

python demo_hang.py &
kill -USR1 $!    # snapshot: the process keeps running
kill $!          # SIGTERM: a second dump, then it exits

The snapshot answers "where is it stuck?" without a debugger (middle trimmed, paths shortened):

reason  : SIGUSR1 snapshot

== THREAD STACKS ==
-- worker-1
  [...]
  File "demo_hang.py", line 18, in worker
    with lock_b:
-- MainThread
  File "demo_hang.py", line 26, in <module>
    with lock_a:
  [...recorder frames omitted...]

== LAST 2 EVENTS (seconds relative to this dump) ==
   -0.974  DEBUG    [worker-1] worker: worker holds lock_a, wants lock_b
   -0.974  DEBUG    [MainThread] worker: main holds lock_b, wants lock_a

The SIGTERM that followed produced the second dump, and the process still exited with status 143 (128 + 15).

install() also points faulthandler at crashdumps/faulthandler.log. SIGUSR2 writes plain Python stacks there at the C level, so it doesn't depend on the interpreter being free to run a Python signal handler, and a hard crash writes its stack there too.

What it costs

I timed 100,000 log.debug("processed item %d of batch %s", ...) calls, best of 5 runs, with a console handler at INFO attached. These are numbers from a small sandbox, so treat them as relative:

Setup

Cost per debug call

Debug disabled (root logger at INFO)

about 0.1 µs

A do-nothing handler at DEBUG

about 4.8 µs

The flight recorder

about 6 µs

DEBUG written to a file on every call

about 8 µs

Most of the recorder's cost is the standard library building the LogRecord, which any DEBUG handler pays; the recorder adds roughly 1.5 µs on top. That is fine for a script or service logging thousands of events a second, and wrong for a tight inner loop logging millions. The file figure is for a local disk where writes land in the OS cache, and it excludes the cost of rotating, shipping and searching that file later.

One more cost is easy to miss: with the root at DEBUG, your libraries' own debug events are recorded too. If one is too chatty, raise it with logging.getLogger("name").setLevel(logging.INFO), knowing its debug events then won't reach the recorder either.

What it can't do

  • It can't survive SIGKILL. The kernel's out-of-memory killer uses it, and no handler runs for it, so no dump is written.

  • Segfaults lose the buffer. In a test that crashed Python with ctypes.string_at(0), no dump was written; only the faulthandler stack survived, headed "Fatal Python error: Segmentation fault".

  • Each process has its own buffer. With multiprocessing or a pre-fork server, make sure every worker ends up with the recorder installed. The pid in each dump's filename keeps them apart.

  • The stack section covers threads, not asyncio tasks. In an async app it shows the event loop's thread, not each pending task.

  • It only knows what you record. A breadcrumb helps only if you thought to write it. The cheapest habit is one note() per loop iteration or request, carrying the identifier you'd want to see in the dump.

  • Windows is untested. SIGUSR1 and SIGUSR2 don't exist there, so install() skips them.

To try it, install the recorder in one script, break something on purpose, and read the dump. That tells you whether the buffer holds what you'd actually want to see before a failure.

Comments(0)

Join the discussion by logging into your account.

No comments yet. Be the first to comment!

Anshu Pathak

Passionate developer sharing knowledge about modern web technologies and best practices.

Subscribe to Anshu Pathak's Newsletter

Direct email dispatches when new stories are published. Zero algorithms.