{"schemaVersion":"1.0","type":"Article","slug":"five-ways-to-invalidate-your-prompt-cache-cgm7k","url":"https://api.zyvop.com/five-ways-to-invalidate-your-prompt-cache-cgm7k","title":"Five ways to invalidate your prompt cache","subtitle":null,"tldr":"Part 2 of 2 on prompt caching. Part 1 covered the economics. Part 1 established the prize: caching cut an 80-turn agent session from $54.08 to $6.91 in my cost ...","keywords":[],"entities":["Lê Đức Minh","AI Engineer","ZyVOP"],"keyTakeaways":["Part 2 of 2 on prompt caching.","Part 1 covered the economics.","Part 1 established the prize: caching cut an 80-turn agent session from $54.08 to $6.91 in my cost model, an 87.2% saving on input tokens."],"headings":["What exactly does the cache match on?","1. A dynamic system prompt","The tool-definition trap","2. History that isn't append-only","3. Compaction","4. Letting the cache go cold (how long is a TTL?)","5. Assuming your provider does it for you","How do you know if your cache is working?","Isn't this what semantic caching does?","The order I'd do this in","TL;DR","Reproduce this","References"],"outboundLinks":["https://zyvop.com/why-your-coding-agent-s-bill-grows-faster-than-the-chat-ig2oh","https://platform.claude.com/docs/en/build-with-claude/prompt-caching","https://ai.google.dev/gemini-api/docs/caching","https://arxiv.org/abs/2309.06180","https://medium.com/@minhle_0210/prompt-context-harness-loop-an-agents-anatomy-642db41429fb","https://www.linkedin.com/in/minhle007/","https://github.com/MinLee0210"],"contentText":"Part 2 of 2 on prompt caching. Part 1 covered the economics. Part 1 established the prize: caching cut an 80-turn agent session from $54.08 to $6.91 in my cost model, an 87.2% saving on input tokens. Once I had that number I got curious about the opposite question — not how much caching saves, but how easy it is to think you have it and not. That turned out to be the more useful half, because every way of losing it is silent. This post is about how to lose it. Not by disabling caching — by writing a harness that looks completely correct and quietly never hits the cache. Every failure below is silent. No error, no warning, no degraded output. The agent works exactly as intended and costs several times more than it should. Prompt caching is a prefix match, not a similarity match. Any change to the earliest part of a request invalidates every cached token after it. In my cost model, putting a timestamp in the system prompt cost 7.8× on an 80-turn session — not because the timestamp is expensive, but because it forfeits the entire discount on every turn. What exactly does the cache match on? Providers cache the longest common prefix between your request and something they recently processed. Prefix. Byte-order, from the front. Not \"a similar request,\" not \"the same information in a different order.\" The comparison walks forward from token zero and stops at the first difference — and everything from that point on is fresh, at full price. Which yields one design principle for the entire harness: Stable content first. Volatile content last. Never change what came before. Every mistake below is a violation of that one line. 1. A dynamic system prompt The most expensive one-line mistake available, and it looks like good engineering. Your system prompt sits at position zero. It's the longest-lived thing in the request. So you put useful context in it: system = f\"\"\"You are a coding assistant. Current time: {datetime.now().isoformat()} Working directory: {os.getcwd()} Available tools: {', '.join(discover_tools())} \"\"\" Every one of those three lines is a cache bomb. The timestamp changes on every call. So the system prompt changes on every call. So the prefix diverges at token ~15 of a 10,000-token system prompt — and every token after it, including your entire conversation history, is billed fresh. Forever. Turns Stable prompt Dynamic prompt Penalty Multiple 10 $0.33 $1.16 $0.83 3.5× 20 $0.79 $3.92 $3.13 4.9× 40 $2.19 $14.24 $12.05 6.5× 80 $6.91 $54.08 $47.17 7.8× A timestamp doesn't cost you a timestamp. It costs you the whole discount, on every turn, for the rest of the session — and the penalty grows with session length, because you're back on the quadratic curve from part 1. The fix is not to drop the information. It's to move it : system = STATIC_PROMPT # never changes, caches beautifully messages = [ {\"role\": \"system\", \"content\": system}, *history, # append-only {\"role\": \"user\", \"content\": f\"<context>time: {now}, cwd: {cwd}</context>\\n{query}\"}, ] Same information, delivered at the end of the request where volatility is free. The model reads it just as well. Watch for the sneaky variants: a tool list assembled by iterating a set (non-deterministic order in some paths), a JSON dump with unsorted keys, a \"user preferences\" block refreshed from a database each call, anything with a request ID. The tool-definition trap Worth its own note, because tool schemas sit immediately after the system prompt — near the very front of the prefix, where instability is most expensive — and they're assembled programmatically, which is exactly where non-determinism creeps in. Three ways I've seen it happen: tools = [schema(t) for t in registry.values()] # dict order: usually fine tools = [schema(t) for t in discovered_plugins] # filesystem order: not fine tools = [schema(t) for t in enabled_tools] # a set: order not guaranteed json.dumps(tool_schema) # key order follows insertion The failure mode is nasty because it's intermittent . A set iterates in a consistent order within one process, so your local test passes. Restart the process, or run on a machine with a different hash seed, and the order shifts — so the cache works fine in development and misses in production, or works for an hour and then stops after a redeploy. Two defences, both cheap. Sort your tool list by name before serialising, and pass sort_keys=True when you dump any JSON that lands in the prompt. Then assert on it: hash the serialised system-prompt-plus-tools block at startup and log it. If that hash changes between two runs that should be identical, you have found your cache leak before it found your invoice. 2. History that isn't append-only The second rule from that design principle, and the one that bites at exactly the wrong moment. Your conversation history must only ever grow at the end. Editing, reordering, or removing an earlier message shifts the divergence point back to wherever you touched — invalidating everything after it. Things that quietly do this: Trimming old messages to stay under the context limit. Dropping the oldest turn changes the prefix at position one and invalidates the entire session cache. You saved context and torched the discount. Re-rendering tool results with fresh formatting, a new timestamp, or a re-serialised payload. Sorting or deduplicating history. Injecting a reminder into the middle of the transcript rather than appending it. The trimming case deserves attention because it's a genuine conflict: you may have to drop messages. Just know that a sliding window over history means you re-pay for the whole transcript every time the window slides. If you must shed context, do it rarely and in large chunks rather than one message per turn. 3. Compaction Compaction — summarising the transcript into something short and continuing — invalidates the cache completely, by design. You've replaced the prefix with a different, shorter prefix. Nothing after position zero matches. This one is not a bug . Compaction is doing exactly what it's supposed to, and the cache reset is the correct price for it. But it should be a deliberate decision rather than a surprise, because the moment you compact you pay a full cache write on the new prefix and start the accumulation curve over. Two practical consequences: Don't compact on a timer. Compact when the context genuinely needs it. Every compaction is a fresh write of everything. Compact in bigger steps, less often. Two compactions cost two cold starts. Same context saved, twice the rewrite. 4. Letting the cache go cold (how long is a TTL?) Caches expire. The TTL varies by provider and by tier — five minutes is a common default, an hour is often available at a higher write cost, and routed or brokered inference may inherit whatever the backend it landed on happens to offer. That means wall-clock time is now a cost variable in your agent , which is an unusual thing to have to think about. Step away for coffee and come back, and your next message pays a full rewrite of the entire transcript. Scenario, 80 turns Cost Extra Uninterrupted $6.91 — One break at turn 40 $7.66 $0.75 Breaks at 20, 40, 60 $9.16 $2.25 A break every 10 turns $12.16 $5.25 Late breaks cost more than early ones, because a cold start rewrites everything so far — and \"everything so far\" is bigger later. A break at turn 70 is far more expensive than the same break at turn 10. This is also the argument for the longer TTL, but only conditionally — as part 1 showed, on an uninterrupted session the 1-hour option is a straight 2× markup on writes for no benefit. It pays at roughly three cold starts. 5. Assuming your provider does it for you The silent one, and the reason to check rather than assume. OpenAI caches eligible prefixes automatically. Anthropic and Gemini require explicit cache markers in the request. Omit them and nothing is cached, forever, with no indication. Conversation-state APIs typically handle it; raw chat-completions calls typically don't. Routers and inference brokers may send consecutive requests to different backends. A warm cache lives on the machine that built it — if you're not pinned, you can miss a cache that exists. Consistent routing is worth real money here. None of these produce an error. The only symptom is the bill. How do you know if your cache is working? Everything above is invisible without instrumentation, so this is the part I'd implement first — before any of the fixes. Log cache-hit rate per request. Every major provider returns cached token counts in the usage block of the response. Read them and record the ratio. The field names differ — Anthropic splits cache_creation_input_tokens from cache_read_input_tokens , OpenAI reports cached_tokens nested inside its prompt token details — so normalise once at your client boundary rather than scattering provider checks through the codebase: def cache_stats(usage): \"\"\"Normalise the usage block into (fresh, written, read).\"\"\" read = getattr(usage, \"cache_read_input_tokens\", 0) or \\ getattr(usage, \"prompt_tokens_details\", {}).get(\"cached_tokens\", 0) written = getattr(usage, \"cache_creation_input_tokens\", 0) total = getattr(usage, \"input_tokens\", 0) or getattr(usage, \"prompt_tokens\", 0) return total - read - written, written, read fresh, written, read = cache_stats(response.usage) log.info(\"cache_hit_rate=%.2f\", read / max(fresh + written + read, 1)) That single ratio is the whole diagnostic. Everything else on this list shows up in it. Put effective price per million on a dashboard. Divide total input spend by total input tokens. On a healthy long session it should sit far below list price — my model landed at $0.51/M against a $4.00 list. If yours hovers near list price, your cache is broken. Watch the shape over a session. A healthy session starts cold, climbs as the transcript grows, and plateaus high. A sawtooth means expiries. A flat line near zero means something in your prefix changes every turn — go and look for a timestamp. Test it deliberately. Send the same two-turn conversation twice and assert that the second turn reports cache hits. It's a cheap test and it catches every failure on this list. Isn't this what semantic caching does? A question that comes up immediately, and the two get conflated constantly, so it's worth separating them. Semantic caching sits in your infrastructure. It embeds an incoming query, looks for a previously answered question that's close enough in vector space, and returns the stored answer without calling the model at all. It caches outputs, it's approximate, and it can be wrong — two questions can be neighbours in embedding space and still want different answers. Prompt caching sits in the provider's infrastructure. It caches the processed state of an input prefix, it's an exact match, and it cannot change what the model returns — only what you're billed for the input. They're complementary, not alternatives. Semantic caching can skip a call entirely, which beats any discount. Prompt caching makes the calls you do make dramatically cheaper. But only one of them can silently give a user a subtly wrong answer, so they deserve very different amounts of scrutiny before you ship them. The order I'd do this in Instrument first. Without a hit-rate number, everything else is guesswork. Check whether your provider caches by default. One doc page; potentially a 7× bill difference. Audit your system prompt for anything volatile. This is the big one, and it's usually a five-minute fix. Verify history is append-only , especially wherever context-limit trimming happens. Only then think about TTL tiers. It's the smallest lever on the list and the only one that costs money to pull. TL;DR The cache is a longest-common-prefix match. Stable content first, volatile content last, never modify what came before. A timestamp in the system prompt cost 7.8× on an 80-turn session in my model — $6.91 becomes $54.08. Move volatile context into the final user message instead. History must be append-only. Sliding-window trimming re-pays for the entire transcript every time the window moves. Compaction resets the cache by design — legitimate, but don't do it on a timer. Anthropic and Gemini don't cache unless you ask. There is no error when you forget; the only symptom is the invoice. Reproduce this python 3.13 · stdlib only system prompt 10,000 tok · user +1,000/turn · assistant +3,000/turn base input $4.00/M · write 1.25x (5-min) / 2.00x (1-hour) · read 0.10x Figures are arithmetic on stated assumptions, not measurements of any provider's billing. References Anthropic — Prompt caching — cache breakpoints, TTL tiers, and the usage fields you need to compute a hit rate. Google — Gemini context caching — a usefully different model: caches are explicit objects you create and reference, which makes the prefix contract impossible to ignore. Kwon et al., Efficient Memory Management for Large Language Model Serving with PagedAttention , SOSP 2023 — explains why the match is a prefix rather than anything cleverer. The KV cache is stored in blocks, and two requests can share a block only while their token sequences are still identical. Everything in this post follows from that. If you build agent harnesses, Prompt, Context, Harness, Loop covers where the harness sits in the first place — the cache lives in exactly the layer that post calls the harness. 👉 Follow me: LinkedIn | GitHub","contentHash":"sha256:694e533e88c8786a983e221e92bc01503d3efca839144294953d34683ae6ddf5","authorName":"Lê Đức Minh","authorUrl":"https://api.zyvop.com/author/l445","authorSameAs":["https://minlee0210.github.io","https://github.com/MinLee0210"],"category":null,"tags":[],"audience":"Readers researching the subject covered by this article","tone":"Professional, ai engineer perspective","readingTimeMinutes":10,"wordCount":2195,"faqs":null,"primaryTopic":null,"publishedAt":"2026-09-02T12:30:01.224Z","updatedAt":"2026-09-02T13:05:44.360Z","canonicalUrl":"https://api.zyvop.com/five-ways-to-invalidate-your-prompt-cache-cgm7k"}