{"schemaVersion":"1.0","type":"TechArticle","types":["Article","TechArticle"],"slug":"when-the-upstream-is-down-serving-stale-data-on-purpose-kdz9c","url":"https://zyvop.com/when-the-upstream-is-down-serving-stale-data-on-purpose-kdz9c","title":"When the Upstream Is Down: Serving Stale Data on Purpose","subtitle":"Your page shouldn’t die, or lie, when a third-party API does. Keep serving the last known good data, show how old it is, and recover on its own.","tldr":"When a dependency goes down, most apps show an error, a blank page, or silently old numbers. This post builds a last-known-good fallback with fresh, stale, and too-old stages, validated responses, back-off, and a visible freshness indicator, plus alerts for when you’re serving stale.","keywords":["resilience","backend","System Design","Caching","graceful-degradation"],"entities":["Anshu Pathak","resilience","backend","System Design","Caching","graceful-degradation","ZyVOP"],"keyTakeaways":["Separate freshness (fresh_for) from retention (max_stale). A TTL that deletes data deletes your fallback.","Store the last good response with its fetched_at, and change that timestamp only on success.","Validate responses before they overwrite good data, and use short timeouts plus back-off with jitter.","Show the age, calmly, and escalate as data gets older. Compute it on the server.","Decide per kind of data: serve stale for display, be careful for decisions, fail closed for money and security.","Persist and warm the store so the fallback exists on a cold start."],"headings":["The goals","Fresh, stale, and too old","The request flow","Implementation","Make the staleness visible","Not all data degrades the same way","The cold-start problem","Know when you're serving stale","Test the outage on purpose","Pitfalls","A quicker option at the HTTP layer","Takeaways"],"outboundLinks":[],"contentText":"Your app shows exchange rates, shipping estimates, weather, or a partner's product catalog, and all of it comes from someone else's API. Then that API has a bad afternoon. What does your app do? Most apps pick one of three behaviors, usually by accident: Show an error page. The upstream is down, so you are down. Your uptime is now the product of every dependency's uptime. Show nothing, or zeros. The call fails, the code catches the exception, and the page renders an empty chart, a blank list, or a total of 0.00. This is worse than an error, because it looks like an answer. Serve old data silently. The page stays up, but nobody can tell the numbers are three hours old. People make decisions on them. The better behavior sits between those: keep serving the last data you know was good, say plainly how old it is, and recover on its own when the upstream returns. This post builds that, including the part most implementations skip: the visible freshness indicator. The goals Stay up. A dependency outage becomes a degraded page, not a dead one. Be honest. Anyone looking at the data can tell it's old, and by how much. Heal automatically. When the upstream recovers, fresh data replaces stale data without a deploy or a manual cache flush. Know about it. You should find out about an outage from your own alerts, not from a user. Fresh, stale, and too old A cached value goes through three stages, controlled by two timers: stateDiagram-v2 [*] --&gt; Fresh: good response stored Fresh --&gt; Stale: age passes fresh_for Stale --&gt; Fresh: refresh succeeds Stale --&gt; TooOld: age passes max_stale TooOld --&gt; Fresh: refresh succeedsFresh (fresh_for, say 5 minutes): serve it with no network call. Stale (up to max_stale, say 6 hours): try to refresh. If the refresh fails, serve the old value and mark it stale. Too old (beyond max_stale): the data is no longer trustworthy enough to show, because \"last known good\" eventually becomes \"last known wrong.\" Return an honest error. The mistake that defeats this whole idea is using an ordinary cache TTL. When a Redis or memcached key expires, the value is deleted, so the moment you need a fallback, it's gone. The fix is to separate how long data counts as fresh from how long you keep it around. Store the value together with fetched_at, never evict it on a timer, and work out freshness when you read it. The request flow sequenceDiagram participant U as Browser participant A as Your app participant S as Last known good store participant P as Upstream API U-&gt;&gt;A: GET /rates A-&gt;&gt;S: read value and fetched_at alt value is fresh S--&gt;&gt;A: value A--&gt;&gt;U: 200 with data, no banner else value is stale or missing A-&gt;&gt;P: fetch with a 2 s timeout alt upstream answers P--&gt;&gt;A: 200 with data A-&gt;&gt;S: store value, fetched_at = now A--&gt;&gt;U: 200 with fresh data else upstream fails or times out A--&gt;&gt;U: 200 with old data, a stale flag, and its age end endIf the store has nothing at all (a cold start with the upstream down), the app has nothing to fall back on and returns a 503. We'll deal with that case later. Implementation This version uses only the Python standard library and SQLite as the store. SQLite is a good fit because the fallback has to survive restarts, and a file on disk does that without another service. The code needs Python 3.10 or newer (for the str | None annotation) and SQLite 3.24 or newer (for the upsert). import json import logging import random import sqlite3 import time import urllib.error import urllib.request from dataclasses import dataclass log = logging.getLogger(\"stale\") # Persistent on purpose: a fallback that lives only in memory disappears on every restart. db = sqlite3.connect(\"stale_cache.db\", isolation_level=None, timeout=10) db.row_factory = sqlite3.Row db.execute(\"PRAGMA journal_mode = WAL\") db.execute(\"\"\" CREATE TABLE IF NOT EXISTS last_known_good ( key TEXT PRIMARY KEY, value TEXT NOT NULL, -- JSON of the last GOOD response fetched_at REAL NOT NULL, -- when that response arrived; only a success changes it failures INTEGER NOT NULL DEFAULT 0, -- consecutive failed refreshes retry_at REAL NOT NULL DEFAULT 0, -- leave a failing upstream alone until then last_error TEXT )\"\"\") class Unavailable(Exception): \"\"\"No usable data: nothing stored, or what we have is older than max_stale.\"\"\" class Gone(Exception): \"\"\"The upstream says the thing does not exist. That is an answer, not an outage.\"\"\" @dataclass class Result: value: object fetched_at: float # unix time of the last good response age: float # seconds since fetched_at stale: bool # True when we could not refresh error: str | None = None # why the refresh failed, if it did def fetch_json(url, timeout=2.0): \"\"\"Example fetcher. Short timeout: a fallback is useless if we wait 30 s before using it.\"\"\" try: with urllib.request.urlopen(url, timeout=timeout) as resp: return json.load(resp) except urllib.error.HTTPError as e: if e.code in (404, 410): raise Gone(url) from e raise # 5xx, 429, 401, 403 ... treat as an outage and fall back def get(key, fetch, fresh_for=300, max_stale=86400, validate=None): now = time.time() row = db.execute(\"SELECT * FROM last_known_good WHERE key = ?\", (key,)).fetchone() if row and now - row[\"fetched_at\"] &lt;= fresh_for: # 1. fresh: no network call at all return Result(json.loads(row[\"value\"]), row[\"fetched_at\"], now - row[\"fetched_at\"], False) usable = row is not None and now - row[\"fetched_at\"] &lt;= max_stale error = None if usable and now &lt; row[\"retry_at\"]: # 2. upstream is in back-off: don't hit it error = row[\"last_error\"] else: # 3. try to refresh try: value = fetch() if validate and not validate(value): raise ValueError(\"response failed validation\") except Gone: raise except Exception as exc: error = f\"{type(exc).__name__}: {exc}\" if row: failures = row[\"failures\"] + 1 delay = min(5 * 2 ** (failures - 1), 300) * random.uniform(0.5, 1.5) db.execute( \"UPDATE last_known_good SET failures = ?, retry_at = ?, last_error = ? WHERE key = ?\", (failures, now + delay, error, key), ) else: db.execute( \"\"\"INSERT INTO last_known_good (key, value, fetched_at) VALUES (?, ?, ?) ON CONFLICT(key) DO UPDATE SET value = excluded.value, fetched_at = excluded.fetched_at, failures = 0, retry_at = 0, last_error = NULL\"\"\", (key, json.dumps(value), now), ) return Result(value, now, 0.0, False) # 4. the refresh failed or was skipped: serve the last known good, if it is not too old if usable: age = now - row[\"fetched_at\"] log.warning(\"serving stale key=%s age=%ds error=%s\", key, age, error) return Result(json.loads(row[\"value\"]), row[\"fetched_at\"], age, True, error) raise Unavailable(f\"{key}: no usable data (upstream error: {error})\")The decisions that matter: fetched_at only changes when a good response arrives. This is the single most important line in the design. If a failed refresh, a stale read, or a cache layer that \"touches\" entries when it serves them reset the timestamp, old data would look fresh and your freshness indicator would lie. A response must be valid before it replaces the last known good. validate catches the case where the upstream returns 200 OK with an error page, an empty object, or a changed schema. A bad response that overwrites good data is how a small outage becomes a long one. The timeout is short. If the upstream hangs and your HTTP client waits 30 seconds, every user waits 30 seconds and then gets the stale data. Pick a timeout that matches what a user will tolerate, usually 1 to 3 seconds. Back-off protects both sides. After a failure the code stops calling the upstream for a growing interval (5 s, 10 s, 20 s, up to 5 minutes, with jitter). Users get the stale value instantly instead of paying for a timeout on every request, the failing service gets breathing room, and recovery doesn't arrive as a synchronized stampede. Back-off only applies while there's something usable to serve. Once the stored value is older than max_stale there's nothing to fall back on, so the code keeps trying and recovers on the first success. Gone is not an outage. A 404 or 410 means the upstream is healthy and says the thing doesn't exist. Serving a stale copy of something that was deleted is a bug, so that case is raised to the caller instead. Not every error is the same kind of error. Timeouts, connection failures, 5xx, and 429 are outages and trigger the fallback. A 401 or 403 means your credentials are broken. Falling back is still the right thing for the user, but it should also page someone, because the fallback will quietly run out of time otherwise. Using it takes a few lines: import functools fetch_rates = functools.partial(fetch_json, \"https://api.example.com/v1/rates\", 2.0) result = get( \"rates:usd\", fetch_rates, fresh_for=300, # 5 minutes counts as fresh max_stale=6 * 3600, # beyond 6 hours, we'd rather show an error validate=lambda v: isinstance(v, dict) and \"rates\" in v, )Make the staleness visible Returning old data is only half of graceful degradation. The other half is telling people. Three places to do it: In the API response. Give clients the facts instead of making them guess: from datetime import datetime, timezone def api_rates(): try: r = get(\"rates:usd\", fetch_rates, fresh_for=300, max_stale=6 * 3600, validate=lambda v: isinstance(v, dict) and \"rates\" in v) except Unavailable: return {\"error\": \"rates_unavailable\"}, 503, {} headers = {\"X-Data-Stale\": \"true\"} if r.stale else {} body = { \"data\": r.value, \"meta\": { \"fetched_at\": datetime.fromtimestamp(r.fetched_at, timezone.utc).isoformat(timespec=\"seconds\"), \"age_seconds\": int(r.age), \"stale\": r.stale, }, } return body, 200, headersIn the UI. A small, calm banner beats a scary one, and it should say the age in words people understand: def humanize_age(seconds): s = int(seconds) if s &lt; 10: return \"just now\" if s &lt; 60: return f\"{s} seconds ago\" for unit, size in ((\"day\", 86400), (\"hour\", 3600), (\"minute\", 60)): if s &gt;= size: n = s // size return f\"{n} {unit}{'' if n == 1 else 's'} ago\"{% if result.stale %} &lt;div class=\"freshness freshness--stale\" role=\"status\"&gt; Showing data from {{ result.age | humanize_age }}. Live data is temporarily unavailable. &lt;/div&gt; {% endif %}This assumes you pass result to the template and register humanize_age as a template filter. Some guidelines for the indicator: Show nothing when data is fresh. A permanent \"updated 2 minutes ago\" label is fine for dashboards, but a banner should mean something is wrong. Show the age, not just the word \"stale.\" \"Showing data from 14 minutes ago\" lets the reader decide whether it still matters. \"Stale data\" doesn't. Put the exact timestamp in a tooltip or title attribute for people who need it. Escalate with age. A neutral note at 15 minutes, a warning color at 2 hours, and disabled actions past whatever threshold is dangerous. Compute age on the server. Send age_seconds, and let the browser add the time elapsed since the page loaded (using something like performance.now()) to keep the label current. Never subtract fetched_at from the browser's clock, because user devices are routinely wrong by minutes or more. Not all data degrades the same way \"Serve stale\" is the right call for a lot of data and the wrong call for some. Decide per kind of data, and write the decision down: Kind of data Examples When it's stale Informational display Weather, news lists, leaderboards, catalog browsing Serve it with a freshness label. A long max_stale is fine. Decision support Stock levels shown to staff, delivery estimates, status dashboards Serve it, label it prominently, and use a short max_stale. Transactional Price at checkout, account balance, inventory reservation Don't guess. Disable the action, or ask the user to confirm the price they're seeing. Security Permissions, revoked tokens, kill switches Don't serve stale beyond a very short window. Default to the safe answer (deny). Stale permissions can grant access that was already taken away. The rule of thumb: the more irreversible the action the data triggers, the less stale it may be. The cold-start problem A last-known-good store only helps if it contains something. Three situations leave it empty: First deploy, or a new key. There's no good data yet and the upstream is already down. A fresh container with no persistent disk. The cache dies with the container. A new region or replica that hasn't served the request before. What helps: Persist the store (a database table, a mounted volume) instead of keeping it in process memory. Warm it on deploy. A startup step or a scheduled job that refreshes the important keys means the fallback exists before it's needed. Ship a baked-in snapshot for the data that matters most: a JSON file committed to the repo, loaded when the store is empty. Label it honestly, for example \"Built-in data from 1 November,\" so even the worst case degrades gracefully. Know when you're serving stale Graceful degradation has a failure mode of its own: it works so well that nobody notices the upstream has been down for four days. Treat \"serving stale\" as an event worth watching. Log it. The code above emits one warning per stale response with the key, the age, and the error. Count these. Expose the age. A health check that lists the oldest data you hold shows trouble quickly: SELECT key, CAST(strftime('%s', 'now') AS INTEGER) - CAST(fetched_at AS INTEGER) AS age_seconds, failures, last_error FROM last_known_good ORDER BY fetched_at LIMIT 10;Alert on age, not on single failures. One failed call is noise. \"Rates haven't refreshed for 30 minutes\" is a signal. Set the alert threshold a little above fresh_for, and a second, louder one well before max_stale, so you hear about it before users see an error. Test the outage on purpose Don't wait for the upstream to fail to find out whether this works. Run a fake upstream locally and break it: Serve a small JSON endpoint and fetch it once. Confirm that the value is stored. Make it return 503. Confirm the app serves the old value, flags it stale, and that age keeps growing rather than resetting. Make it hang for longer than your timeout. Confirm responses still arrive in about one timeout, not the full hang. Call it repeatedly while it's down. Confirm the upstream is not contacted during back-off. Bring it back. Confirm the next refresh succeeds, the stale flag clears, and failures returns to zero. Return a 200 with a garbage body. Confirm the last known good value is untouched. Return 404. Confirm you get Gone, not the old data. Let the data age past max_stale. Confirm you get a clean error, not ancient data. Those eight checks cover the real failure modes: hard failure, slow failure, flapping, bad data, and permanent absence. Pitfalls urlopen's timeout isn't a deadline. In many HTTP clients, including the standard library's, the timeout applies to each blocking socket operation. A server that trickles bytes slowly can keep you waiting well beyond it. If you need a hard cap on total time, use a client that supports an overall deadline, or enforce one yourself. Caching per user without keying by user. If the data differs by user or tenant, the key must include that. Otherwise the fallback shows one customer's data to another. A thundering herd when the entry expires. If 200 requests arrive while a refresh is due, 200 requests hit the upstream. Use a per-key lock within one process, or a lease row across processes, so one caller refreshes while the rest serve the stale value. A quicker option at the HTTP layer If your data flows through an HTTP cache or CDN, two Cache-Control extensions from RFC 5861 do part of this for you: stale-while-revalidate (serve stale while refreshing in the background) and stale-if-error (serve stale when the origin returns errors). Support varies between CDNs and proxies, so check the documentation for the one you use. They're a good first step, but they don't replace the application-level version. They can't distinguish a 404 from an outage in the way your code can, they don't validate payloads, and they don't give your UI the age of the data. Many setups use both: headers at the edge, and a last-known-good store in the app for the data that matters. You can also serve stale while refreshing in the background inside your own app: return the old value immediately and kick off the refresh on a worker thread or a queue. It lowers latency further, at the cost of more moving parts, so start with the synchronous version above. Takeaways Separate freshness (fresh_for) from retention (max_stale). A TTL that deletes data deletes your fallback. Store the last good response with its fetched_at, and change that timestamp only on success. Validate responses before they overwrite good data, and use short timeouts plus back-off with jitter. Show the age, calmly, and escalate as data gets older. Compute it on the server. Decide per kind of data: serve stale for display, be careful for decisions, fail closed for money and security. Persist and warm the store so the fallback exists on a cold start. Alert on how long you've been stale, and rehearse the outage before it happens.","contentHash":"sha256:b48eb825e424725159b547aed9d8d46d2ef90a1cfa8eb5428bfb6203ffa306f0","authorName":"Anshu Pathak","authorUrl":"https://zyvop.com/author/anshu","authorSameAs":["https://www.linkedin.com/in/anshu-pathak-zyvop/"],"category":"System Design","tags":["resilience","backend","System Design","Caching","graceful-degradation"],"audience":"Software engineers and developers building applications with System Design","tone":"Practical and evidence-based engineering guidance","readingTimeMinutes":11,"wordCount":2797,"faqs":null,"primaryTopic":"System Design","publishedAt":"2026-11-18T04:55:41.060Z","updatedAt":"2026-11-18T04:55:41.060Z","canonicalUrl":"https://zyvop.com/when-the-upstream-is-down-serving-stale-data-on-purpose-kdz9c"}