--- title: "Save Debug Logs Only When Your Python Script Crashes" author: "Anshu Pathak" published: "2026-11-14T12:31:13.682Z" updated: "2026-11-14T12:31:13.682Z" url: "https://zyvop.com/save-debug-logs-only-when-your-python-script-crashes-f3zvj" canonical: "https://zyvop.com/save-debug-logs-only-when-your-python-script-crashes-f3zvj" tags: ["devtools","Python","debugging","logging","observability"] --- # Save Debug Logs Only When Your Python Script Crashes An in-memory ring buffer of recent debug events, written to disk only on a crash, a hang or a logged error, in under 200 lines of standard-library Python. ![Cover image]() Log at INFO and the output stays readable, but the line you need after a failure is almost always a DEBUG line you didn't keep. Log at DEBUG and you keep it, along with a flood of everything else: more disk, more noise to search through, and more chances of writing a secret into a file that lives forever. Aircraft solve the same problem with a flight recorder: record continuously, keep only the recent past, and read it only after something goes wrong. The idea carries over to a script or a small service almost unchanged. This post builds one as a standard `logging` handler: no dependencies, no changes to your existing log calls, and under 200 lines. ## How it works The recorder is a ring buffer, a `deque` with a maximum length, fed by a logging handler that sees every level. When it fills up, the oldest events fall off the end. A `note()` method adds breadcrumbs for things your log calls don't cover. Nothing touches the disk until a trigger fires: | Trigger | Why it exists | | --- | --- | | Unhandled exception, in the main thread or any other | The classic crash. Ctrl-C arrives the same way, which is handy for interrupting a hung loop. | | A logged `ERROR`, at most one dump a minute by default | Services usually catch exceptions and log them, so the unhandled-exception hook never fires. | | `SIGTERM` | A supervisor or a timeout is killing the process, and this is its last chance to explain itself. | | `SIGUSR1` | Snapshot a process that looks stuck without killing it. | A dump file holds the reason, the traceback if there is one, the stack of every thread, and the last N events with timestamps relative to the moment of the dump. ```mermaid flowchart LR A["Your code
logging calls and note()"] --> B[("Ring buffer
last N events")] A --> C["Console or file handler
INFO and up"] T{"Any trigger
fires"} --> D["Dump file"] B -.->|read at dump time| D ``` ### Why not the standard library's `MemoryHandler`? `logging.handlers.MemoryHandler` looks like the obvious fit, but when its buffer fills up it flushes everything to its target instead of discarding the oldest records. In a quick test with a capacity of 3 and 7 debug records, the target received 6 of them. It is a batching tool, so every record eventually reaches the disk, which is exactly what we're trying to avoid. ## The code Save this as `recorder.py`. It needs Python 3.8 or newer (for `threading.excepthook`); I ran everything here on 3.12 on Linux. ```python import collections import faulthandler import logging import os import platform import re import signal import sys import threading import time import traceback # Best-effort scrubbing, applied when a dump is written (never to the live buffer). # The part wrapped in (?P...) survives; everything after it is replaced. DEFAULT_REDACT = ( r"(?i)(?P\b(?:authorization|password|passwd|secret|token|api[-_]?key)\b" r"[\"']?\s*[:=]\s*[\"']?(?:(?:Bearer|Basic|Digest|Token)\s+)?)[^\s,;&\"']+", ) class FlightRecorder(logging.Handler): def __init__(self, directory="crashdumps", capacity=500, max_dumps=10, max_msg=500, dump_on=logging.ERROR, cooldown=60, redact=DEFAULT_REDACT): super().__init__(level=logging.DEBUG) self.directory, self.max_dumps, self.max_msg = directory, max_dumps, max_msg self.dump_on, self.cooldown = dump_on, cooldown self.redact = [re.compile(p) for p in redact] self.events = collections.deque(maxlen=capacity) # oldest events fall off the end self._lock = threading.RLock() # re-entrant: a signal can land while we hold it self._last_dump = 0.0 self._count = 0 # ---- recording ------------------------------------------------------- def emit(self, record): try: try: msg = record.getMessage() except Exception: msg = f"" exc = record.exc_info if record.exc_info and record.exc_info[0] else None if exc: msg += f" !! {exc[0].__name__}: {exc[1]}" self._add(record.created, record.levelname, record.name, record.threadName, msg) if self.dump_on is not None and record.levelno >= self.dump_on: self.dump(f"{record.levelname} logged by {record.name}", exc, throttle=True) except Exception: self.handleError(record) def note(self, event, **fields): """A breadcrumb: record a step or a value that logging calls don't cover.""" parts = [event] for key, value in fields.items(): try: parts.append(f"{key}={value!r}") except Exception: parts.append(f"{key}=") self._add(time.time(), "NOTE", "breadcrumb", threading.current_thread().name, " ".join(parts)) def _add(self, ts, level, name, thread, msg): if len(msg) > self.max_msg: # store text, not objects: bounded and can't change later msg = msg[:self.max_msg] + f"...[{len(msg) - self.max_msg} more chars]" with self._lock: self.events.append((ts, level, name, thread, msg)) # ---- dumping --------------------------------------------------------- def dump(self, reason, exc_info=None, throttle=False): """Write the buffer to a file. Never raises; returns the path or None.""" try: now = time.time() with self._lock: if throttle and now - self._last_dump < self.cooldown: return None self._last_dump = now self._count += 1 n, events = self._count, list(self.events) out = [f"reason : {reason}", f"time : {time.strftime('%Y-%m-%d %H:%M:%S %z', time.localtime(now))}", f"process : pid {os.getpid()}, argv {sys.argv}", f"python : {platform.python_version()} on {platform.platform()}"] if exc_info: out += ["", "== EXCEPTION =="] out += "".join(traceback.format_exception(*exc_info)).rstrip("\n").split("\n") out += ["", "== THREAD STACKS =="] names = {t.ident: t.name for t in threading.enumerate()} for ident, frame in sys._current_frames().items(): out.append(f"-- {names.get(ident, ident)}") out += "".join(traceback.format_stack(frame)).rstrip("\n").split("\n") out += ["", f"== LAST {len(events)} EVENTS (seconds relative to this dump) =="] out += [f"{ts - now:+9.3f} {lvl:<8} [{thread}] {name}: {msg}" for ts, lvl, name, thread, msg in events] text = self._scrub("\n".join(out)) + "\n" os.makedirs(self.directory, mode=0o700, exist_ok=True) stamp = time.strftime("%Y%m%d-%H%M%S", time.localtime(now)) path = os.path.join(self.directory, f"dump-{stamp}-{os.getpid()}-{n}.txt") fd = os.open(path, os.O_WRONLY | os.O_CREAT | os.O_EXCL, 0o600) # owner-only with os.fdopen(fd, "w", encoding="utf-8") as f: f.write(text) self._prune() print(f"flight recorder: wrote {path}", file=sys.stderr) return path except Exception as e: print(f"flight recorder: dump failed: {e!r}", file=sys.stderr) return None def _scrub(self, text): for rx in self.redact: text = rx.sub(lambda m: (m.groupdict().get("keep") or "") + "[REDACTED]", text) return text def _prune(self): dumps = sorted((os.path.join(self.directory, f) for f in os.listdir(self.directory) if f.startswith("dump-")), key=os.path.getmtime) for old in dumps[:-self.max_dumps]: os.remove(old) # ---- wiring ---------------------------------------------------------- def install(self): """Attach to the root logger and hook crashes and signals. Call it once, from the main thread, after your own logging and signal setup.""" root = logging.getLogger() previous = root.level or logging.WARNING for handler in root.handlers: # keep existing handlers as quiet as they were if handler.level == logging.NOTSET: handler.setLevel(previous) root.setLevel(logging.DEBUG) # the recorder now sees everything root.addHandler(self) prev_hook = sys.excepthook def on_crash(etype, value, tb): self.dump(f"unhandled {etype.__name__}", (etype, value, tb)) prev_hook(etype, value, tb) sys.excepthook = on_crash prev_thread_hook = threading.excepthook def on_thread_crash(args): if args.exc_type is not SystemExit: who = args.thread.name if args.thread else "unknown thread" self.dump(f"unhandled {args.exc_type.__name__} in {who}", (args.exc_type, args.exc_value, args.exc_traceback)) prev_thread_hook(args) threading.excepthook = on_thread_crash os.makedirs(self.directory, mode=0o700, exist_ok=True) self._fault_log = open(os.path.join(self.directory, "faulthandler.log"), "a") faulthandler.enable(self._fault_log) # hard crashes: stacks only, the buffer is lost if hasattr(faulthandler, "register") and hasattr(signal, "SIGUSR2"): faulthandler.register(signal.SIGUSR2, file=self._fault_log) # works even if Python is stuck in C prev_term = signal.getsignal(signal.SIGTERM) def on_term(signum, frame): self.dump("SIGTERM received") if callable(prev_term): prev_term(signum, frame) # let the app shut down its own way elif prev_term != signal.SIG_IGN: signal.signal(signum, signal.SIG_DFL) os.kill(os.getpid(), signum) # die with the usual exit status signal.signal(signal.SIGTERM, on_term) if hasattr(signal, "SIGUSR1"): # live snapshot of a stuck process, no kill needed signal.signal(signal.SIGUSR1, lambda signum, frame: self.dump("SIGUSR1 snapshot")) return self ``` A few decisions are worth explaining: - **Existing handlers keep their level.** Raising the root logger to DEBUG would make a console handler with no level of its own start printing every debug line. `install()` first pins such handlers to the previous root level. Call it after `logging.basicConfig(...)`, because a handler added later with no level will show debug output. - **Events are stored as truncated text, not objects.** The buffer can't keep large objects alive, and a message can't change after the fact. At the defaults the full buffer held about 330 KB in my test, with every slot at the maximum message length. - **The lock is an** `RLock` **for a reason.** Python runs signal handlers on the main thread between bytecodes, which can be while that thread is inside `_add()` holding the lock. With a plain `Lock`, the handler waits forever for a lock its own thread holds. I tested this by firing 200 `SIGUSR1` signals during heavy logging: the `RLock` version finished, and the plain-`Lock` version hung until I killed it. - `SIGTERM` **chains instead of replacing.** After dumping, it calls whatever handler your app had installed. If there was none, it re-delivers the default signal, so the process still exits as "killed by signal 15" rather than with a made-up exit code. - **Dumps are treated as sensitive.** Files are owner-only (`0600`) in a `0700` directory, only the newest 10 are kept, and secrets are scrubbed at dump time by a regex list. That list covers `password=...`, `Authorization: Bearer ...` and `api_key: ...` shapes, and it is best effort, so add patterns for whatever your logs contain. ## Using it ```python import logging from recorder import FlightRecorder logging.basicConfig(level=logging.INFO) # your normal setup, unchanged rec = FlightRecorder(capacity=500).install() rec.note("batch start", batch_id=batch.id, size=len(batch)) # optional breadcrumbs ``` Call `install()` once, from the main thread, because Python only allows signal handlers to be set there. ## Crash test This script imports pages of rows from a pretend feed. Page 4 contains one row with no price, and the code doesn't expect that. ```python import logging import time from recorder import FlightRecorder logging.basicConfig(level=logging.INFO, format="%(asctime)s %(levelname)s %(message)s") rec = FlightRecorder(capacity=200).install() log = logging.getLogger("importer") def fetch_page(n): log.debug("GET /feed?page=%d", n) time.sleep(0.001) # pretend network rows = [{"id": n * 100 + i, "price": f"{1000 + i * 37:,}.00", "currency": "USD"} for i in range(5)] if n == 4: rows[3]["price"] = None # the one bad row in the feed log.debug("page %d -> %d rows", n, len(rows)) return rows def parse(row): log.debug("parsing row %r", row) return float(row["price"].replace(",", "")) total = 0.0 for page in range(1, 8): rec.note("page start", page=page, total=round(total, 2)) for row in fetch_page(page): total += parse(row) log.info("page %d done, running total %.2f", page, total) ``` With the console at INFO, this is everything you get (paths shortened): ```text 2026-11-14 08:03:30,534 INFO page 1 done, running total 5370.00 2026-11-14 08:03:30,535 INFO page 2 done, running total 10740.00 2026-11-14 08:03:30,537 INFO page 3 done, running total 16110.00 flight recorder: wrote crashdumps/dump-20261114-080330-96-1.txt Traceback (most recent call last): File "demo_crash.py", line 31, in total += parse(row) ^^^^^^^^^^ File "demo_crash.py", line 24, in parse return float(row["price"].replace(",", "")) ^^^^^^^^^^^^^^^^^^^^ AttributeError: 'NoneType' object has no attribute 'replace' ``` The traceback says what failed but not which row, and nothing logged at INFO can tell you. The dump can (excerpt, with the middle trimmed): ```text reason : unhandled AttributeError time : 2026-11-14 08:03:30 +0000 process : pid 96, argv ['demo_crash.py'] == EXCEPTION == [same traceback as above] == LAST 34 EVENTS (seconds relative to this dump) == [...] -0.001 NOTE [MainThread] breadcrumb: page start page=4 total=16110.0 -0.001 DEBUG [MainThread] importer: GET /feed?page=4 -0.000 DEBUG [MainThread] importer: page 4 -> 5 rows -0.000 DEBUG [MainThread] importer: parsing row {'id': 400, 'price': '1,000.00', 'currency': 'USD'} -0.000 DEBUG [MainThread] importer: parsing row {'id': 401, 'price': '1,037.00', 'currency': 'USD'} -0.000 DEBUG [MainThread] importer: parsing row {'id': 402, 'price': '1,074.00', 'currency': 'USD'} -0.000 DEBUG [MainThread] importer: parsing row {'id': 403, 'price': None, 'currency': 'USD'} ``` The offending row is the last event: `id` 403, `price` of `None`. The `-0.000` offsets are an artifact of this toy script running in milliseconds; in a real service they show how long before the failure each event happened. ## Hang test Crashes are the easy case. The harder one is a process that is still running but doing nothing. This script deadlocks on purpose: the main thread holds `lock_b` and wants `lock_a`, while a worker holds `lock_a` and wants `lock_b`. ```python import logging import threading import time from recorder import FlightRecorder logging.basicConfig(level=logging.INFO, format="%(asctime)s %(levelname)s %(message)s") FlightRecorder().install() log = logging.getLogger("worker") lock_a, lock_b = threading.Lock(), threading.Lock() def worker(): with lock_a: log.debug("worker holds lock_a, wants lock_b") time.sleep(0.2) with lock_b: log.info("worker done") threading.Thread(target=worker, name="worker-1", daemon=True).start() with lock_b: log.debug("main holds lock_b, wants lock_a") time.sleep(0.2) with lock_a: log.info("main done") ``` Run it in the background, ask for a snapshot, then stop it: ```bash python demo_hang.py & kill -USR1 $! # snapshot: the process keeps running kill $! # SIGTERM: a second dump, then it exits ``` The snapshot answers "where is it stuck?" without a debugger (middle trimmed, paths shortened): ```text reason : SIGUSR1 snapshot == THREAD STACKS == -- worker-1 [...] File "demo_hang.py", line 18, in worker with lock_b: -- MainThread File "demo_hang.py", line 26, in with lock_a: [...recorder frames omitted...] == LAST 2 EVENTS (seconds relative to this dump) == -0.974 DEBUG [worker-1] worker: worker holds lock_a, wants lock_b -0.974 DEBUG [MainThread] worker: main holds lock_b, wants lock_a ``` The `SIGTERM` that followed produced the second dump, and the process still exited with status 143 (128 + 15). `install()` also points `faulthandler` at `crashdumps/faulthandler.log`. `SIGUSR2` writes plain Python stacks there at the C level, so it doesn't depend on the interpreter being free to run a Python signal handler, and a hard crash writes its stack there too. ## What it costs I timed 100,000 `log.debug("processed item %d of batch %s", ...)` calls, best of 5 runs, with a console handler at INFO attached. These are numbers from a small sandbox, so treat them as relative: | Setup | Cost per debug call | | --- | --- | | Debug disabled (root logger at INFO) | about 0.1 µs | | A do-nothing handler at DEBUG | about 4.8 µs | | The flight recorder | about 6 µs | | DEBUG written to a file on every call | about 8 µs | Most of the recorder's cost is the standard library building the `LogRecord`, which any DEBUG handler pays; the recorder adds roughly 1.5 µs on top. That is fine for a script or service logging thousands of events a second, and wrong for a tight inner loop logging millions. The file figure is for a local disk where writes land in the OS cache, and it excludes the cost of rotating, shipping and searching that file later. One more cost is easy to miss: with the root at DEBUG, your libraries' own debug events are recorded too. If one is too chatty, raise it with `logging.getLogger("name").setLevel(logging.INFO)`, knowing its debug events then won't reach the recorder either. ## What it can't do - **It can't survive** `SIGKILL`**.** The kernel's out-of-memory killer uses it, and no handler runs for it, so no dump is written. - **Segfaults lose the buffer.** In a test that crashed Python with `ctypes.string_at(0)`, no dump was written; only the `faulthandler` stack survived, headed "Fatal Python error: Segmentation fault". - **Each process has its own buffer.** With `multiprocessing` or a pre-fork server, make sure every worker ends up with the recorder installed. The pid in each dump's filename keeps them apart. - **The stack section covers threads, not asyncio tasks.** In an async app it shows the event loop's thread, not each pending task. - **It only knows what you record.** A breadcrumb helps only if you thought to write it. The cheapest habit is one `note()` per loop iteration or request, carrying the identifier you'd want to see in the dump. - **Windows is untested.** `SIGUSR1` and `SIGUSR2` don't exist there, so `install()` skips them. To try it, install the recorder in one script, break something on purpose, and read the dump. That tells you whether the buffer holds what you'd actually want to see before a failure.