
Log at INFO and the output stays readable, but the line you need after a failure is almost always a DEBUG line you didn't keep. Log at DEBUG and you keep it, along with a flood of everything else: more disk, more noise to search through, and more chances of writing a secret into a file that lives forever.
Aircraft solve the same problem with a flight recorder: record continuously, keep only the recent past, and read it only after something goes wrong. The idea carries over to a script or a small service almost unchanged. This post builds one as a standard logging handler: no dependencies, no changes to your existing log calls, and under 200 lines.
How it works
The recorder is a ring buffer, a deque with a maximum length, fed by a logging handler that sees every level. When it fills up, the oldest events fall off the end. A note() method adds breadcrumbs for things your log calls don't cover. Nothing touches the disk until a trigger fires:
Trigger | Why it exists |
|---|---|
Unhandled exception, in the main thread or any other | The classic crash. Ctrl-C arrives the same way, which is handy for interrupting a hung loop. |
A logged | Services usually catch exceptions and log them, so the unhandled-exception hook never fires. |
| A supervisor or a timeout is killing the process, and this is its last chance to explain itself. |
| Snapshot a process that looks stuck without killing it. |
A dump file holds the reason, the traceback if there is one, the stack of every thread, and the last N events with timestamps relative to the moment of the dump.
flowchart LR
A["Your code<br/>logging calls and note()"] --> B[("Ring buffer<br/>last N events")]
A --> C["Console or file handler<br/>INFO and up"]
T{"Any trigger<br/>fires"} --> D["Dump file"]
B -.->|read at dump time| DWhy not the standard library's MemoryHandler?
logging.handlers.MemoryHandler looks like the obvious fit, but when its buffer fills up it flushes everything to its target instead of discarding the oldest records. In a quick test with a capacity of 3 and 7 debug records, the target received 6 of them. It is a batching tool, so every record eventually reaches the disk, which is exactly what we're trying to avoid.
The code
Save this as recorder.py. It needs Python 3.8 or newer (for threading.excepthook); I ran everything here on 3.12 on Linux.
import collections
import faulthandler
import logging
import os
import platform
import re
import signal
import sys
import threading
import time
import traceback
# Best-effort scrubbing, applied when a dump is written (never to the live buffer).
# The part wrapped in (?P<keep>...) survives; everything after it is replaced.
DEFAULT_REDACT = (
r"(?i)(?P<keep>\b(?:authorization|password|passwd|secret|token|api[-_]?key)\b"
r"[\"']?\s*[:=]\s*[\"']?(?:(?:Bearer|Basic|Digest|Token)\s+)?)[^\s,;&\"']+",
)
class FlightRecorder(logging.Handler):
def __init__(self, directory="crashdumps", capacity=500, max_dumps=10,
max_msg=500, dump_on=logging.ERROR, cooldown=60, redact=DEFAULT_REDACT):
super().__init__(level=logging.DEBUG)
self.directory, self.max_dumps, self.max_msg = directory, max_dumps, max_msg
self.dump_on, self.cooldown = dump_on, cooldown
self.redact = [re.compile(p) for p in redact]
self.events = collections.deque(maxlen=capacity) # oldest events fall off the end
self._lock = threading.RLock() # re-entrant: a signal can land while we hold it
self._last_dump = 0.0
self._count = 0
# ---- recording -------------------------------------------------------
def emit(self, record):
try:
try:
msg = record.getMessage()
except Exception:
msg = f"<unformattable message: {record.msg!r}>"
exc = record.exc_info if record.exc_info and record.exc_info[0] else None
if exc:
msg += f" !! {exc[0].__name__}: {exc[1]}"
self._add(record.created, record.levelname, record.name, record.threadName, msg)
if self.dump_on is not None and record.levelno >= self.dump_on:
self.dump(f"{record.levelname} logged by {record.name}", exc, throttle=True)
except Exception:
self.handleError(record)
def note(self, event, **fields):
"""A breadcrumb: record a step or a value that logging calls don't cover."""
parts = [event]
for key, value in fields.items():
try:
parts.append(f"{key}={value!r}")
except Exception:
parts.append(f"{key}=<unrepresentable>")
self._add(time.time(), "NOTE", "breadcrumb", threading.current_thread().name, " ".join(parts))
def _add(self, ts, level, name, thread, msg):
if len(msg) > self.max_msg: # store text, not objects: bounded and can't change later
msg = msg[:self.max_msg] + f"...[{len(msg) - self.max_msg} more chars]"
with self._lock:
self.events.append((ts, level, name, thread, msg))
# ---- dumping ---------------------------------------------------------
def dump(self, reason, exc_info=None, throttle=False):
"""Write the buffer to a file. Never raises; returns the path or None."""
try:
now = time.time()
with self._lock:
if throttle and now - self._last_dump < self.cooldown:
return None
self._last_dump = now
self._count += 1
n, events = self._count, list(self.events)
out = [f"reason : {reason}",
f"time : {time.strftime('%Y-%m-%d %H:%M:%S %z', time.localtime(now))}",
f"process : pid {os.getpid()}, argv {sys.argv}",
f"python : {platform.python_version()} on {platform.platform()}"]
if exc_info:
out += ["", "== EXCEPTION =="]
out += "".join(traceback.format_exception(*exc_info)).rstrip("\n").split("\n")
out += ["", "== THREAD STACKS =="]
names = {t.ident: t.name for t in threading.enumerate()}
for ident, frame in sys._current_frames().items():
out.append(f"-- {names.get(ident, ident)}")
out += "".join(traceback.format_stack(frame)).rstrip("\n").split("\n")
out += ["", f"== LAST {len(events)} EVENTS (seconds relative to this dump) =="]
out += [f"{ts - now:+9.3f} {lvl:<8} [{thread}] {name}: {msg}"
for ts, lvl, name, thread, msg in events]
text = self._scrub("\n".join(out)) + "\n"
os.makedirs(self.directory, mode=0o700, exist_ok=True)
stamp = time.strftime("%Y%m%d-%H%M%S", time.localtime(now))
path = os.path.join(self.directory, f"dump-{stamp}-{os.getpid()}-{n}.txt")
fd = os.open(path, os.O_WRONLY | os.O_CREAT | os.O_EXCL, 0o600) # owner-only
with os.fdopen(fd, "w", encoding="utf-8") as f:
f.write(text)
self._prune()
print(f"flight recorder: wrote {path}", file=sys.stderr)
return path
except Exception as e:
print(f"flight recorder: dump failed: {e!r}", file=sys.stderr)
return None
def _scrub(self, text):
for rx in self.redact:
text = rx.sub(lambda m: (m.groupdict().get("keep") or "") + "[REDACTED]", text)
return text
def _prune(self):
dumps = sorted((os.path.join(self.directory, f) for f in os.listdir(self.directory)
if f.startswith("dump-")), key=os.path.getmtime)
for old in dumps[:-self.max_dumps]:
os.remove(old)
# ---- wiring ----------------------------------------------------------
def install(self):
"""Attach to the root logger and hook crashes and signals. Call it once,
from the main thread, after your own logging and signal setup."""
root = logging.getLogger()
previous = root.level or logging.WARNING
for handler in root.handlers: # keep existing handlers as quiet as they were
if handler.level == logging.NOTSET:
handler.setLevel(previous)
root.setLevel(logging.DEBUG) # the recorder now sees everything
root.addHandler(self)
prev_hook = sys.excepthook
def on_crash(etype, value, tb):
self.dump(f"unhandled {etype.__name__}", (etype, value, tb))
prev_hook(etype, value, tb)
sys.excepthook = on_crash
prev_thread_hook = threading.excepthook
def on_thread_crash(args):
if args.exc_type is not SystemExit:
who = args.thread.name if args.thread else "unknown thread"
self.dump(f"unhandled {args.exc_type.__name__} in {who}",
(args.exc_type, args.exc_value, args.exc_traceback))
prev_thread_hook(args)
threading.excepthook = on_thread_crash
os.makedirs(self.directory, mode=0o700, exist_ok=True)
self._fault_log = open(os.path.join(self.directory, "faulthandler.log"), "a")
faulthandler.enable(self._fault_log) # hard crashes: stacks only, the buffer is lost
if hasattr(faulthandler, "register") and hasattr(signal, "SIGUSR2"):
faulthandler.register(signal.SIGUSR2, file=self._fault_log) # works even if Python is stuck in C
prev_term = signal.getsignal(signal.SIGTERM)
def on_term(signum, frame):
self.dump("SIGTERM received")
if callable(prev_term):
prev_term(signum, frame) # let the app shut down its own way
elif prev_term != signal.SIG_IGN:
signal.signal(signum, signal.SIG_DFL)
os.kill(os.getpid(), signum) # die with the usual exit status
signal.signal(signal.SIGTERM, on_term)
if hasattr(signal, "SIGUSR1"): # live snapshot of a stuck process, no kill needed
signal.signal(signal.SIGUSR1, lambda signum, frame: self.dump("SIGUSR1 snapshot"))
return selfA few decisions are worth explaining:
Existing handlers keep their level. Raising the root logger to DEBUG would make a console handler with no level of its own start printing every debug line.
install()first pins such handlers to the previous root level. Call it afterlogging.basicConfig(...), because a handler added later with no level will show debug output.Events are stored as truncated text, not objects. The buffer can't keep large objects alive, and a message can't change after the fact. At the defaults the full buffer held about 330 KB in my test, with every slot at the maximum message length.
The lock is an
RLockfor a reason. Python runs signal handlers on the main thread between bytecodes, which can be while that thread is inside_add()holding the lock. With a plainLock, the handler waits forever for a lock its own thread holds. I tested this by firing 200SIGUSR1signals during heavy logging: theRLockversion finished, and the plain-Lockversion hung until I killed it.SIGTERMchains instead of replacing. After dumping, it calls whatever handler your app had installed. If there was none, it re-delivers the default signal, so the process still exits as "killed by signal 15" rather than with a made-up exit code.Dumps are treated as sensitive. Files are owner-only (
0600) in a0700directory, only the newest 10 are kept, and secrets are scrubbed at dump time by a regex list. That list coverspassword=...,Authorization: Bearer ...andapi_key: ...shapes, and it is best effort, so add patterns for whatever your logs contain.
Using it
import logging
from recorder import FlightRecorder
logging.basicConfig(level=logging.INFO) # your normal setup, unchanged
rec = FlightRecorder(capacity=500).install()
rec.note("batch start", batch_id=batch.id, size=len(batch)) # optional breadcrumbsCall install() once, from the main thread, because Python only allows signal handlers to be set there.
Crash test
This script imports pages of rows from a pretend feed. Page 4 contains one row with no price, and the code doesn't expect that.
import logging
import time
from recorder import FlightRecorder
logging.basicConfig(level=logging.INFO, format="%(asctime)s %(levelname)s %(message)s")
rec = FlightRecorder(capacity=200).install()
log = logging.getLogger("importer")
def fetch_page(n):
log.debug("GET /feed?page=%d", n)
time.sleep(0.001) # pretend network
rows = [{"id": n * 100 + i, "price": f"{1000 + i * 37:,}.00", "currency": "USD"}
for i in range(5)]
if n == 4:
rows[3]["price"] = None # the one bad row in the feed
log.debug("page %d -> %d rows", n, len(rows))
return rows
def parse(row):
log.debug("parsing row %r", row)
return float(row["price"].replace(",", ""))
total = 0.0
for page in range(1, 8):
rec.note("page start", page=page, total=round(total, 2))
for row in fetch_page(page):
total += parse(row)
log.info("page %d done, running total %.2f", page, total)With the console at INFO, this is everything you get (paths shortened):
2026-11-14 08:03:30,534 INFO page 1 done, running total 5370.00
2026-11-14 08:03:30,535 INFO page 2 done, running total 10740.00
2026-11-14 08:03:30,537 INFO page 3 done, running total 16110.00
flight recorder: wrote crashdumps/dump-20261114-080330-96-1.txt
Traceback (most recent call last):
File "demo_crash.py", line 31, in <module>
total += parse(row)
^^^^^^^^^^
File "demo_crash.py", line 24, in parse
return float(row["price"].replace(",", ""))
^^^^^^^^^^^^^^^^^^^^
AttributeError: 'NoneType' object has no attribute 'replace'The traceback says what failed but not which row, and nothing logged at INFO can tell you. The dump can (excerpt, with the middle trimmed):
reason : unhandled AttributeError
time : 2026-11-14 08:03:30 +0000
process : pid 96, argv ['demo_crash.py']
== EXCEPTION ==
[same traceback as above]
== LAST 34 EVENTS (seconds relative to this dump) ==
[...]
-0.001 NOTE [MainThread] breadcrumb: page start page=4 total=16110.0
-0.001 DEBUG [MainThread] importer: GET /feed?page=4
-0.000 DEBUG [MainThread] importer: page 4 -> 5 rows
-0.000 DEBUG [MainThread] importer: parsing row {'id': 400, 'price': '1,000.00', 'currency': 'USD'}
-0.000 DEBUG [MainThread] importer: parsing row {'id': 401, 'price': '1,037.00', 'currency': 'USD'}
-0.000 DEBUG [MainThread] importer: parsing row {'id': 402, 'price': '1,074.00', 'currency': 'USD'}
-0.000 DEBUG [MainThread] importer: parsing row {'id': 403, 'price': None, 'currency': 'USD'}The offending row is the last event: id 403, price of None. The -0.000 offsets are an artifact of this toy script running in milliseconds; in a real service they show how long before the failure each event happened.
Hang test
Crashes are the easy case. The harder one is a process that is still running but doing nothing. This script deadlocks on purpose: the main thread holds lock_b and wants lock_a, while a worker holds lock_a and wants lock_b.
import logging
import threading
import time
from recorder import FlightRecorder
logging.basicConfig(level=logging.INFO, format="%(asctime)s %(levelname)s %(message)s")
FlightRecorder().install()
log = logging.getLogger("worker")
lock_a, lock_b = threading.Lock(), threading.Lock()
def worker():
with lock_a:
log.debug("worker holds lock_a, wants lock_b")
time.sleep(0.2)
with lock_b:
log.info("worker done")
threading.Thread(target=worker, name="worker-1", daemon=True).start()
with lock_b:
log.debug("main holds lock_b, wants lock_a")
time.sleep(0.2)
with lock_a:
log.info("main done")Run it in the background, ask for a snapshot, then stop it:
python demo_hang.py &
kill -USR1 $! # snapshot: the process keeps running
kill $! # SIGTERM: a second dump, then it exitsThe snapshot answers "where is it stuck?" without a debugger (middle trimmed, paths shortened):
reason : SIGUSR1 snapshot
== THREAD STACKS ==
-- worker-1
[...]
File "demo_hang.py", line 18, in worker
with lock_b:
-- MainThread
File "demo_hang.py", line 26, in <module>
with lock_a:
[...recorder frames omitted...]
== LAST 2 EVENTS (seconds relative to this dump) ==
-0.974 DEBUG [worker-1] worker: worker holds lock_a, wants lock_b
-0.974 DEBUG [MainThread] worker: main holds lock_b, wants lock_aThe SIGTERM that followed produced the second dump, and the process still exited with status 143 (128 + 15).
install() also points faulthandler at crashdumps/faulthandler.log. SIGUSR2 writes plain Python stacks there at the C level, so it doesn't depend on the interpreter being free to run a Python signal handler, and a hard crash writes its stack there too.
What it costs
I timed 100,000 log.debug("processed item %d of batch %s", ...) calls, best of 5 runs, with a console handler at INFO attached. These are numbers from a small sandbox, so treat them as relative:
Setup | Cost per debug call |
|---|---|
Debug disabled (root logger at INFO) | about 0.1 µs |
A do-nothing handler at DEBUG | about 4.8 µs |
The flight recorder | about 6 µs |
DEBUG written to a file on every call | about 8 µs |
Most of the recorder's cost is the standard library building the LogRecord, which any DEBUG handler pays; the recorder adds roughly 1.5 µs on top. That is fine for a script or service logging thousands of events a second, and wrong for a tight inner loop logging millions. The file figure is for a local disk where writes land in the OS cache, and it excludes the cost of rotating, shipping and searching that file later.
One more cost is easy to miss: with the root at DEBUG, your libraries' own debug events are recorded too. If one is too chatty, raise it with logging.getLogger("name").setLevel(logging.INFO), knowing its debug events then won't reach the recorder either.
What it can't do
It can't survive
SIGKILL. The kernel's out-of-memory killer uses it, and no handler runs for it, so no dump is written.Segfaults lose the buffer. In a test that crashed Python with
ctypes.string_at(0), no dump was written; only thefaulthandlerstack survived, headed "Fatal Python error: Segmentation fault".Each process has its own buffer. With
multiprocessingor a pre-fork server, make sure every worker ends up with the recorder installed. The pid in each dump's filename keeps them apart.The stack section covers threads, not asyncio tasks. In an async app it shows the event loop's thread, not each pending task.
It only knows what you record. A breadcrumb helps only if you thought to write it. The cheapest habit is one
note()per loop iteration or request, carrying the identifier you'd want to see in the dump.Windows is untested.
SIGUSR1andSIGUSR2don't exist there, soinstall()skips them.
To try it, install the recorder in one script, break something on purpose, and read the dump. That tells you whether the buffer holds what you'd actually want to see before a failure.
Comments(0)
Join the discussion by logging into your account.
No comments yet. Be the first to comment!