{"schemaVersion":"1.0","type":"TechArticle","types":["Article","TechArticle"],"slug":"save-debug-logs-only-when-your-python-script-crashes-f3zvj","url":"https://zyvop.com/save-debug-logs-only-when-your-python-script-crashes-f3zvj","title":"Save Debug Logs Only When Your Python Script Crashes","subtitle":"An in-memory ring buffer of recent debug events, written to disk only on a crash, a hang or a logged error, in under 200 lines of standard-library Python.","tldr":"INFO logs are quiet but miss the detail you need when something breaks; DEBUG logs keep it but drown you. A small in-memory buffer gives you both: it writes recent events and every thread's stack to a file only when something goes wrong. Includes full code, a crash test and a deadlock test.","keywords":["devtools","Python","debugging","logging","observability"],"entities":["Anshu Pathak","devtools","Python","debugging","logging","observability","ZyVOP"],"keyTakeaways":["Log at INFO and the output stays readable, but the line you need after a failure is almost always a DEBUG line you didn't keep.","Log at DEBUG and you keep it, along with a flood of everything else: more disk, more noise to search through, and more chances of writing a secret into a file that lives forever.","Aircraft solve the same problem with a flight recorder: record continuously, keep only the recent past, and read it only after something goes wrong."],"headings":["How it works","Why not the standard library's MemoryHandler?","The code","Using it","Crash test","Hang test","What it costs","What it can't do"],"outboundLinks":[],"contentText":"Log at INFO and the output stays readable, but the line you need after a failure is almost always a DEBUG line you didn't keep. Log at DEBUG and you keep it, along with a flood of everything else: more disk, more noise to search through, and more chances of writing a secret into a file that lives forever. Aircraft solve the same problem with a flight recorder: record continuously, keep only the recent past, and read it only after something goes wrong. The idea carries over to a script or a small service almost unchanged. This post builds one as a standard logging handler: no dependencies, no changes to your existing log calls, and under 200 lines. How it works The recorder is a ring buffer, a deque with a maximum length, fed by a logging handler that sees every level. When it fills up, the oldest events fall off the end. A note() method adds breadcrumbs for things your log calls don't cover. Nothing touches the disk until a trigger fires: Trigger Why it exists Unhandled exception, in the main thread or any other The classic crash. Ctrl-C arrives the same way, which is handy for interrupting a hung loop. A logged ERROR, at most one dump a minute by default Services usually catch exceptions and log them, so the unhandled-exception hook never fires. SIGTERM A supervisor or a timeout is killing the process, and this is its last chance to explain itself. SIGUSR1 Snapshot a process that looks stuck without killing it. A dump file holds the reason, the traceback if there is one, the stack of every thread, and the last N events with timestamps relative to the moment of the dump. flowchart LR A[\"Your code&lt;br/&gt;logging calls and note()\"] --&gt; B[(\"Ring buffer&lt;br/&gt;last N events\")] A --&gt; C[\"Console or file handler&lt;br/&gt;INFO and up\"] T{\"Any trigger&lt;br/&gt;fires\"} --&gt; D[\"Dump file\"] B -.-&gt;|read at dump time| DWhy not the standard library's MemoryHandler? logging.handlers.MemoryHandler looks like the obvious fit, but when its buffer fills up it flushes everything to its target instead of discarding the oldest records. In a quick test with a capacity of 3 and 7 debug records, the target received 6 of them. It is a batching tool, so every record eventually reaches the disk, which is exactly what we're trying to avoid. The code Save this as recorder.py. It needs Python 3.8 or newer (for threading.excepthook); I ran everything here on 3.12 on Linux. import collections import faulthandler import logging import os import platform import re import signal import sys import threading import time import traceback # Best-effort scrubbing, applied when a dump is written (never to the live buffer). # The part wrapped in (?P&lt;keep&gt;...) survives; everything after it is replaced. DEFAULT_REDACT = ( r\"(?i)(?P&lt;keep&gt;\\b(?:authorization|password|passwd|secret|token|api[-_]?key)\\b\" r\"[\\\"']?\\s*[:=]\\s*[\\\"']?(?:(?:Bearer|Basic|Digest|Token)\\s+)?)[^\\s,;&amp;\\\"']+\", ) class FlightRecorder(logging.Handler): def __init__(self, directory=\"crashdumps\", capacity=500, max_dumps=10, max_msg=500, dump_on=logging.ERROR, cooldown=60, redact=DEFAULT_REDACT): super().__init__(level=logging.DEBUG) self.directory, self.max_dumps, self.max_msg = directory, max_dumps, max_msg self.dump_on, self.cooldown = dump_on, cooldown self.redact = [re.compile(p) for p in redact] self.events = collections.deque(maxlen=capacity) # oldest events fall off the end self._lock = threading.RLock() # re-entrant: a signal can land while we hold it self._last_dump = 0.0 self._count = 0 # ---- recording ------------------------------------------------------- def emit(self, record): try: try: msg = record.getMessage() except Exception: msg = f\"&lt;unformattable message: {record.msg!r}&gt;\" exc = record.exc_info if record.exc_info and record.exc_info[0] else None if exc: msg += f\" !! {exc[0].__name__}: {exc[1]}\" self._add(record.created, record.levelname, record.name, record.threadName, msg) if self.dump_on is not None and record.levelno &gt;= self.dump_on: self.dump(f\"{record.levelname} logged by {record.name}\", exc, throttle=True) except Exception: self.handleError(record) def note(self, event, **fields): \"\"\"A breadcrumb: record a step or a value that logging calls don't cover.\"\"\" parts = [event] for key, value in fields.items(): try: parts.append(f\"{key}={value!r}\") except Exception: parts.append(f\"{key}=&lt;unrepresentable&gt;\") self._add(time.time(), \"NOTE\", \"breadcrumb\", threading.current_thread().name, \" \".join(parts)) def _add(self, ts, level, name, thread, msg): if len(msg) &gt; self.max_msg: # store text, not objects: bounded and can't change later msg = msg[:self.max_msg] + f\"...[{len(msg) - self.max_msg} more chars]\" with self._lock: self.events.append((ts, level, name, thread, msg)) # ---- dumping --------------------------------------------------------- def dump(self, reason, exc_info=None, throttle=False): \"\"\"Write the buffer to a file. Never raises; returns the path or None.\"\"\" try: now = time.time() with self._lock: if throttle and now - self._last_dump &lt; self.cooldown: return None self._last_dump = now self._count += 1 n, events = self._count, list(self.events) out = [f\"reason : {reason}\", f\"time : {time.strftime('%Y-%m-%d %H:%M:%S %z', time.localtime(now))}\", f\"process : pid {os.getpid()}, argv {sys.argv}\", f\"python : {platform.python_version()} on {platform.platform()}\"] if exc_info: out += [\"\", \"== EXCEPTION ==\"] out += \"\".join(traceback.format_exception(*exc_info)).rstrip(\"\\n\").split(\"\\n\") out += [\"\", \"== THREAD STACKS ==\"] names = {t.ident: t.name for t in threading.enumerate()} for ident, frame in sys._current_frames().items(): out.append(f\"-- {names.get(ident, ident)}\") out += \"\".join(traceback.format_stack(frame)).rstrip(\"\\n\").split(\"\\n\") out += [\"\", f\"== LAST {len(events)} EVENTS (seconds relative to this dump) ==\"] out += [f\"{ts - now:+9.3f} {lvl:&lt;8} [{thread}] {name}: {msg}\" for ts, lvl, name, thread, msg in events] text = self._scrub(\"\\n\".join(out)) + \"\\n\" os.makedirs(self.directory, mode=0o700, exist_ok=True) stamp = time.strftime(\"%Y%m%d-%H%M%S\", time.localtime(now)) path = os.path.join(self.directory, f\"dump-{stamp}-{os.getpid()}-{n}.txt\") fd = os.open(path, os.O_WRONLY | os.O_CREAT | os.O_EXCL, 0o600) # owner-only with os.fdopen(fd, \"w\", encoding=\"utf-8\") as f: f.write(text) self._prune() print(f\"flight recorder: wrote {path}\", file=sys.stderr) return path except Exception as e: print(f\"flight recorder: dump failed: {e!r}\", file=sys.stderr) return None def _scrub(self, text): for rx in self.redact: text = rx.sub(lambda m: (m.groupdict().get(\"keep\") or \"\") + \"[REDACTED]\", text) return text def _prune(self): dumps = sorted((os.path.join(self.directory, f) for f in os.listdir(self.directory) if f.startswith(\"dump-\")), key=os.path.getmtime) for old in dumps[:-self.max_dumps]: os.remove(old) # ---- wiring ---------------------------------------------------------- def install(self): \"\"\"Attach to the root logger and hook crashes and signals. Call it once, from the main thread, after your own logging and signal setup.\"\"\" root = logging.getLogger() previous = root.level or logging.WARNING for handler in root.handlers: # keep existing handlers as quiet as they were if handler.level == logging.NOTSET: handler.setLevel(previous) root.setLevel(logging.DEBUG) # the recorder now sees everything root.addHandler(self) prev_hook = sys.excepthook def on_crash(etype, value, tb): self.dump(f\"unhandled {etype.__name__}\", (etype, value, tb)) prev_hook(etype, value, tb) sys.excepthook = on_crash prev_thread_hook = threading.excepthook def on_thread_crash(args): if args.exc_type is not SystemExit: who = args.thread.name if args.thread else \"unknown thread\" self.dump(f\"unhandled {args.exc_type.__name__} in {who}\", (args.exc_type, args.exc_value, args.exc_traceback)) prev_thread_hook(args) threading.excepthook = on_thread_crash os.makedirs(self.directory, mode=0o700, exist_ok=True) self._fault_log = open(os.path.join(self.directory, \"faulthandler.log\"), \"a\") faulthandler.enable(self._fault_log) # hard crashes: stacks only, the buffer is lost if hasattr(faulthandler, \"register\") and hasattr(signal, \"SIGUSR2\"): faulthandler.register(signal.SIGUSR2, file=self._fault_log) # works even if Python is stuck in C prev_term = signal.getsignal(signal.SIGTERM) def on_term(signum, frame): self.dump(\"SIGTERM received\") if callable(prev_term): prev_term(signum, frame) # let the app shut down its own way elif prev_term != signal.SIG_IGN: signal.signal(signum, signal.SIG_DFL) os.kill(os.getpid(), signum) # die with the usual exit status signal.signal(signal.SIGTERM, on_term) if hasattr(signal, \"SIGUSR1\"): # live snapshot of a stuck process, no kill needed signal.signal(signal.SIGUSR1, lambda signum, frame: self.dump(\"SIGUSR1 snapshot\")) return selfA few decisions are worth explaining: Existing handlers keep their level. Raising the root logger to DEBUG would make a console handler with no level of its own start printing every debug line. install() first pins such handlers to the previous root level. Call it after logging.basicConfig(...), because a handler added later with no level will show debug output. Events are stored as truncated text, not objects. The buffer can't keep large objects alive, and a message can't change after the fact. At the defaults the full buffer held about 330 KB in my test, with every slot at the maximum message length. The lock is an RLock for a reason. Python runs signal handlers on the main thread between bytecodes, which can be while that thread is inside _add() holding the lock. With a plain Lock, the handler waits forever for a lock its own thread holds. I tested this by firing 200 SIGUSR1 signals during heavy logging: the RLock version finished, and the plain-Lock version hung until I killed it. SIGTERM chains instead of replacing. After dumping, it calls whatever handler your app had installed. If there was none, it re-delivers the default signal, so the process still exits as \"killed by signal 15\" rather than with a made-up exit code. Dumps are treated as sensitive. Files are owner-only (0600) in a 0700 directory, only the newest 10 are kept, and secrets are scrubbed at dump time by a regex list. That list covers password=..., Authorization: Bearer ... and api_key: ... shapes, and it is best effort, so add patterns for whatever your logs contain. Using it import logging from recorder import FlightRecorder logging.basicConfig(level=logging.INFO) # your normal setup, unchanged rec = FlightRecorder(capacity=500).install() rec.note(\"batch start\", batch_id=batch.id, size=len(batch)) # optional breadcrumbsCall install() once, from the main thread, because Python only allows signal handlers to be set there. Crash test This script imports pages of rows from a pretend feed. Page 4 contains one row with no price, and the code doesn't expect that. import logging import time from recorder import FlightRecorder logging.basicConfig(level=logging.INFO, format=\"%(asctime)s %(levelname)s %(message)s\") rec = FlightRecorder(capacity=200).install() log = logging.getLogger(\"importer\") def fetch_page(n): log.debug(\"GET /feed?page=%d\", n) time.sleep(0.001) # pretend network rows = [{\"id\": n * 100 + i, \"price\": f\"{1000 + i * 37:,}.00\", \"currency\": \"USD\"} for i in range(5)] if n == 4: rows[3][\"price\"] = None # the one bad row in the feed log.debug(\"page %d -&gt; %d rows\", n, len(rows)) return rows def parse(row): log.debug(\"parsing row %r\", row) return float(row[\"price\"].replace(\",\", \"\")) total = 0.0 for page in range(1, 8): rec.note(\"page start\", page=page, total=round(total, 2)) for row in fetch_page(page): total += parse(row) log.info(\"page %d done, running total %.2f\", page, total)With the console at INFO, this is everything you get (paths shortened): 2026-11-14 08:03:30,534 INFO page 1 done, running total 5370.00 2026-11-14 08:03:30,535 INFO page 2 done, running total 10740.00 2026-11-14 08:03:30,537 INFO page 3 done, running total 16110.00 flight recorder: wrote crashdumps/dump-20261114-080330-96-1.txt Traceback (most recent call last): File \"demo_crash.py\", line 31, in &lt;module&gt; total += parse(row) ^^^^^^^^^^ File \"demo_crash.py\", line 24, in parse return float(row[\"price\"].replace(\",\", \"\")) ^^^^^^^^^^^^^^^^^^^^ AttributeError: 'NoneType' object has no attribute 'replace'The traceback says what failed but not which row, and nothing logged at INFO can tell you. The dump can (excerpt, with the middle trimmed): reason : unhandled AttributeError time : 2026-11-14 08:03:30 +0000 process : pid 96, argv ['demo_crash.py'] == EXCEPTION == [same traceback as above] == LAST 34 EVENTS (seconds relative to this dump) == [...] -0.001 NOTE [MainThread] breadcrumb: page start page=4 total=16110.0 -0.001 DEBUG [MainThread] importer: GET /feed?page=4 -0.000 DEBUG [MainThread] importer: page 4 -&gt; 5 rows -0.000 DEBUG [MainThread] importer: parsing row {'id': 400, 'price': '1,000.00', 'currency': 'USD'} -0.000 DEBUG [MainThread] importer: parsing row {'id': 401, 'price': '1,037.00', 'currency': 'USD'} -0.000 DEBUG [MainThread] importer: parsing row {'id': 402, 'price': '1,074.00', 'currency': 'USD'} -0.000 DEBUG [MainThread] importer: parsing row {'id': 403, 'price': None, 'currency': 'USD'}The offending row is the last event: id 403, price of None. The -0.000 offsets are an artifact of this toy script running in milliseconds; in a real service they show how long before the failure each event happened. Hang test Crashes are the easy case. The harder one is a process that is still running but doing nothing. This script deadlocks on purpose: the main thread holds lock_b and wants lock_a, while a worker holds lock_a and wants lock_b. import logging import threading import time from recorder import FlightRecorder logging.basicConfig(level=logging.INFO, format=\"%(asctime)s %(levelname)s %(message)s\") FlightRecorder().install() log = logging.getLogger(\"worker\") lock_a, lock_b = threading.Lock(), threading.Lock() def worker(): with lock_a: log.debug(\"worker holds lock_a, wants lock_b\") time.sleep(0.2) with lock_b: log.info(\"worker done\") threading.Thread(target=worker, name=\"worker-1\", daemon=True).start() with lock_b: log.debug(\"main holds lock_b, wants lock_a\") time.sleep(0.2) with lock_a: log.info(\"main done\")Run it in the background, ask for a snapshot, then stop it: python demo_hang.py &amp; kill -USR1 $! # snapshot: the process keeps running kill $! # SIGTERM: a second dump, then it exitsThe snapshot answers \"where is it stuck?\" without a debugger (middle trimmed, paths shortened): reason : SIGUSR1 snapshot == THREAD STACKS == -- worker-1 [...] File \"demo_hang.py\", line 18, in worker with lock_b: -- MainThread File \"demo_hang.py\", line 26, in &lt;module&gt; with lock_a: [...recorder frames omitted...] == LAST 2 EVENTS (seconds relative to this dump) == -0.974 DEBUG [worker-1] worker: worker holds lock_a, wants lock_b -0.974 DEBUG [MainThread] worker: main holds lock_b, wants lock_aThe SIGTERM that followed produced the second dump, and the process still exited with status 143 (128 + 15). install() also points faulthandler at crashdumps/faulthandler.log. SIGUSR2 writes plain Python stacks there at the C level, so it doesn't depend on the interpreter being free to run a Python signal handler, and a hard crash writes its stack there too. What it costs I timed 100,000 log.debug(\"processed item %d of batch %s\", ...) calls, best of 5 runs, with a console handler at INFO attached. These are numbers from a small sandbox, so treat them as relative: Setup Cost per debug call Debug disabled (root logger at INFO) about 0.1 µs A do-nothing handler at DEBUG about 4.8 µs The flight recorder about 6 µs DEBUG written to a file on every call about 8 µs Most of the recorder's cost is the standard library building the LogRecord, which any DEBUG handler pays; the recorder adds roughly 1.5 µs on top. That is fine for a script or service logging thousands of events a second, and wrong for a tight inner loop logging millions. The file figure is for a local disk where writes land in the OS cache, and it excludes the cost of rotating, shipping and searching that file later. One more cost is easy to miss: with the root at DEBUG, your libraries' own debug events are recorded too. If one is too chatty, raise it with logging.getLogger(\"name\").setLevel(logging.INFO), knowing its debug events then won't reach the recorder either. What it can't do It can't survive SIGKILL. The kernel's out-of-memory killer uses it, and no handler runs for it, so no dump is written. Segfaults lose the buffer. In a test that crashed Python with ctypes.string_at(0), no dump was written; only the faulthandler stack survived, headed \"Fatal Python error: Segmentation fault\". Each process has its own buffer. With multiprocessing or a pre-fork server, make sure every worker ends up with the recorder installed. The pid in each dump's filename keeps them apart. The stack section covers threads, not asyncio tasks. In an async app it shows the event loop's thread, not each pending task. It only knows what you record. A breadcrumb helps only if you thought to write it. The cheapest habit is one note() per loop iteration or request, carrying the identifier you'd want to see in the dump. Windows is untested. SIGUSR1 and SIGUSR2 don't exist there, so install() skips them. To try it, install the recorder in one script, break something on purpose, and read the dump. That tells you whether the buffer holds what you'd actually want to see before a failure.","contentHash":"sha256:2b82b1987dbe00e164f7bfed5679e88b3154af3786c312d150fd8a018fbf4b58","authorName":"Anshu Pathak","authorUrl":"https://zyvop.com/author/anshu","authorSameAs":["https://www.linkedin.com/in/anshu-pathak-zyvop/"],"category":null,"tags":["devtools","Python","debugging","logging","observability"],"audience":"Software engineers and developers building applications with devtools","tone":"Practical and evidence-based engineering guidance","readingTimeMinutes":11,"wordCount":2680,"faqs":null,"primaryTopic":"devtools","publishedAt":"2026-11-14T12:31:13.682Z","updatedAt":"2026-11-14T12:31:13.682Z","canonicalUrl":"https://zyvop.com/save-debug-logs-only-when-your-python-script-crashes-f3zvj"}