Cutting LLM API Costs: Prompt Caching, Batching and Smaller Models

Your prototype cost a few dollars a day. Then it got real users, and the invoice looked like a typo.

That's normal. LLM costs scale with tokens, and tokens pile up quickly: long prompts sent again on every request, chatty outputs, and a top-tier model doing work a cheaper one could handle.

The good news is that three changes do most of the work. Prompt caching makes repeated input cheap. Batching halves the price of work that can wait. Smaller models handle the easy requests for a fraction of the cost.

This post covers each one with real prices, code that runs, and the mistakes that quietly cancel the savings. The numbers come from Anthropic's pricing and API docs. The ideas carry over to other providers, and there's a short comparison near the end.

Where the money goes

You pay per token, and input and output tokens have different prices. Here are the current Claude API rates, per million tokens:

Haiku 5.5 prices shown are for prompts up to 100,000 tokens, and cached tokens count toward that length. Longer prompts cost more, at $0.50 in and $2.50 out.

Three things jump out:

• Output costs 5x input on every model here. Long answers are expensive.

• A cache read costs a fraction of normal input. For Sonnet 5.5 it's $0.10 instead of $2.

• Haiku 5.5 is 20x cheaper than Sonnet 5.5, on both input and output.

Which lever matters depends on the shape of your workload. A support bot with a huge instruction prompt and tiny answers is an input problem. A report writer is an output problem.

Here's how to choose:

flowchart TD
    A["A workload that costs too much"] --> B{"Is a user waiting for the answer?"}
    B -->|"No"| C["Batch it: 50% off"]
    B -->|"Yes"| D{"Does every request start with the same big prefix?"}
    C --> D
    D -->|"Yes"| E["Cache the prefix: reads cost 5 to 10% of normal"]
    D -->|"No"| F{"Is the task easy?"}
    E --> F
    F -->|"Yes"| G["Use a smaller model or lower effort"]
    F -->|"Not sure"| H["Cascade: cheap model first, escalate on failure"]
    F -->|"No"| I["Keep the big model and trim tokens"]

Measure before you cut

You can't fix what you can't see. Every API response reports its token usage, so you can price each request as it happens.

This small module does that. It also holds the cost model we'll use for the rest of the post:

from dataclasses import dataclass


@dataclass(frozen=True)
class Price:
    """Dollars per million tokens."""

    input: float
    output: float
    cache_write: float  # 5-minute cache writes
    cache_read: float


# From Anthropic's pricing page, checked 29 Oct 2026. Prices change, so re-check before you rely on them.
# Haiku 5.5 shows its rate for prompts up to 100,000 tokens (longer prompts cost more).
PRICES = {
    "claude-opus-5-5": Price(input=4.00, output=20.00, cache_write=5.00, cache_read=0.20),
    "claude-sonnet-5-5": Price(input=2.00, output=10.00, cache_write=2.50, cache_read=0.10),
    "claude-haiku-5-5": Price(input=0.10, output=0.50, cache_write=0.125, cache_read=0.01),
}
BATCH_MULTIPLIER = 0.5  # the Batch API charges 50% of the standard price


def request_cost(model: str, usage, batch: bool = False) -> float:
    """Dollar cost of one response. `usage` is the usage object from the API response."""
    p = PRICES[model]
    dollars = (
        usage.input_tokens * p.input
        + (usage.cache_creation_input_tokens or 0) * p.cache_write  # these two can be None
        + (usage.cache_read_input_tokens or 0) * p.cache_read       # when caching isn't used
        + usage.output_tokens * p.output
    ) / 1_000_000
    return dollars * (BATCH_MULTIPLIER if batch else 1)


def cache_hit_rate(usage) -> float:
    """Share of this request's input tokens that were read from the cache."""
    read = usage.cache_read_input_tokens or 0
    total = usage.input_tokens + (usage.cache_creation_input_tokens or 0) + read
    return read / total if total else 0.0


@dataclass(frozen=True)
class Workload:
    requests: int       # per month
    prefix_tokens: int  # identical on every request: instructions, examples, tool definitions
    new_tokens: int     # different every time: the user's input
    output_tokens: int


def monthly_cost(model: str, w: Workload, cache_hit_share=None, batch: bool = False) -> float:
    """cache_hit_share is the fraction of requests that read the prefix from cache.
    Leave it as None for no caching. The rest pay the 1.25x cache write."""
    p = PRICES[model]
    if cache_hit_share is None:
        input_cost = w.requests * (w.prefix_tokens + w.new_tokens) * p.input
    else:
        hits = w.requests * cache_hit_share
        misses = w.requests - hits
        input_cost = (
            hits * w.prefix_tokens * p.cache_read
            + misses * w.prefix_tokens * p.cache_write
            + w.requests * w.new_tokens * p.input
        )
    output_cost = w.requests * w.output_tokens * p.output
    return (input_cost + output_cost) / 1_000_000 * (BATCH_MULTIPLIER if batch else 1)


def cascade_cost(cheap: str, strong: str, w: Workload, escalation_rate: float,
                 cache_hit_share=None, batch: bool = False) -> float:
    """Every request goes to the cheap model. The share that fails validation is sent to the strong one too."""
    escalated = Workload(round(w.requests * escalation_rate), w.prefix_tokens, w.new_tokens, w.output_tokens)
    return (monthly_cost(cheap, w, cache_hit_share, batch)
            + monthly_cost(strong, escalated, cache_hit_share, batch))

request_cost turns a response's usage into dollars. cache_hit_rate tells you how much of the input came from the cache. monthly_cost and cascade_cost let you ask "what if" before you change anything.

Log these two numbers for every request, and you'll quickly see where your money goes.

Lever 1: Prompt caching

Most prompts are mostly the same. The system instructions, the examples, the tool definitions and the reference document don't change from one request to the next. Only the user's input does.

Prompt caching lets the API reuse the work it already did on that repeated part. You pay a small premium once to write it, then a fraction of the normal price to read it.

How it works

The API caches a prefix of your prompt, in this order: tools, then system, then messages. Anything before the breakpoint is cached. Anything after it is processed fresh each time.

So the rule is simple. Put what stays the same first, and what changes last:

flowchart LR
    subgraph Stable["Stable prefix: cache this"]
        T["Tool definitions"] --> S["System instructions"] --> E["Examples and reference docs"]
    end
    subgraph Changes["Changes on every request"]
        U["User input, timestamps, per-request data"]
    end
    E -->|"cache_control goes here"| U

The prices work like this, as multipliers on the normal input price:

The 5-minute cache refreshes for free every time it's used. So steady traffic keeps it warm indefinitely. Anthropic notes that the 5-minute write pays for itself after a single read.

Turn it on

The simplest way is automatic caching: add one cache_control field at the top level of the request. The system caches up to the last cacheable block and moves the breakpoint forward as a conversation grows.

When your prompt has a stable prefix and a changing end, which is the usual case for classifiers and agents, mark the end of the stable part explicitly:

import anthropic
from costs import cache_hit_rate, request_cost

client = anthropic.Anthropic()
INSTRUCTIONS = "...your long, stable instructions and examples..."

response = client.messages.create(
    model="claude-sonnet-5-5",
    max_tokens=20,
    system=[{"type": "text", "text": INSTRUCTIONS, "cache_control": {"type": "ephemeral"}}],
    messages=[{"role": "user", "content": ticket_text}],
)

print(f"${request_cost('claude-sonnet-5-5', response.usage):.5f}")
print(f"{cache_hit_rate(response.usage):.0%} of input came from the cache")

The first call writes the cache. Calls within the next five minutes read it. Check the usage fields to see which one you got:

• cache_creation_input_tokens is what was written

• cache_read_input_tokens is what was read

• input_tokens is only what came after the last breakpoint

Add all three for your real input total. A common surprise is a small input_tokens number that looks wrong but is actually the sign that caching works.

The traps

Most caching problems are one of these:

• The prefix is too short. For the current models the minimum is 512 tokens. Below that, the request still works, but nothing is cached and there's no error. If both cache fields read zero, check the length.

• The breakpoint sits on something that changes. Put cache_control on a block holding a timestamp or the user's message, and every request writes a new entry that nothing ever reads. You pay the 1.25x write each time and never get a hit. Put it on the last block that stays identical.

• Something in the prefix changed. Editing tool definitions, adding or removing images, changing tool_choice, or changing the thinking or effort settings invalidates cached content. Set these once per workload instead of flipping them per request.

• Key order isn't stable. Some languages, Swift and Go among them, shuffle object keys when serializing JSON. If that happens to the keys in your tool_use blocks, a different order is a different prefix, and the cache misses.

• Parallel first requests. A cache entry only becomes available once the first response starts. Fire 50 requests at once on a cold cache and none of them can hit it. Send one first, and wait for its response to begin.

• The gap is longer than five minutes. The clock starts when a request starts, not when it finishes. For traffic that comes in bursts with long pauses, use the 1-hour cache at 2x the write price. It pays off after two reads.

And one that cuts the other way: caching isn't free if it never hits. In the example later in this post, a cache with no hits makes the bill about 21% higher. It starts to pay once roughly one request in five reads from the cache.

Lever 2: Batching

If nobody is waiting for the answer, you shouldn't pay the real-time price.

The Message Batches API takes a pile of requests and processes them asynchronously. Everything is billed at 50% of the standard price, input and output.

Where it fits

Batching suits work like:

• Running evals over thousands of test cases

• Classifying or tagging an archive of old records

• Nightly summaries and reports

• Content moderation sweeps

• Backfills after a prompt change

It doesn't suit anything interactive. A user staring at a spinner can't wait for a batch.

The limits

• A batch holds up to 100,000 requests or 256 MB, whichever comes first.

• Most batches finish within an hour, but the limit is 24 hours. Requests that haven't run by then expire.

• Requests that error, are canceled or expire aren't billed.

• Results stay available for 29 days.

• Results come back in any order, so match them by custom_id.

Here's the flow. This is adapted from the examples in Anthropic's docs:

import time

from anthropic.types.message_create_params import MessageCreateParamsNonStreaming
from anthropic.types.messages.batch_create_params import Request

SYSTEM = [{"type": "text", "text": INSTRUCTIONS, "cache_control": {"type": "ephemeral", "ttl": "1h"}}]

batch = client.messages.batches.create(
    requests=[
        Request(
            custom_id=f"ticket-{i}",
            params=MessageCreateParamsNonStreaming(
                model="claude-haiku-5-5",
                max_tokens=10,
                system=SYSTEM,
                messages=[{"role": "user", "content": text}],
            ),
        )
        for i, text in enumerate(tickets)
    ]
)

while client.messages.batches.retrieve(batch.id).processing_status != "ended":
    time.sleep(60)

for item in client.messages.batches.results(batch.id):
    if item.result.type == "succeeded":
        print(item.custom_id, item.result.message.content[0].text)

Before you submit thousands of requests, test one with the normal Messages API. Batch validation errors only show up after the whole batch has been processed.

Batching and caching together

The two discounts stack. But there's a catch: batch requests run concurrently and in any order, so cache hits are best effort. Anthropic says typical hit rates range from 30% to 98%.

Three habits raise the odds:

1. Use identical cache_control blocks in every request.

2. Use the 1-hour cache, since batches often take longer than five minutes.

3. Write the cache first. Submit one request with the shared prefix, wait for it to finish, then submit the rest.

Not asynchronous, but still cheaper

Some providers also sell a cheaper tier that stays synchronous. Google's Gemini API documents a Flex tier at 50% off, served from off-peak capacity. It's worth knowing about when you can tolerate slower or less predictable responses but don't want to rework your code around batch jobs.

Lever 3: Smaller models

This is the biggest lever and the riskiest one. If the smaller model does the job, the savings are huge. If it doesn't, you've bought a cheaper way to be wrong.

Anthropic's own guidance is to use Haiku for simple tasks, Sonnet for most production work, and Opus for the hardest reasoning. Haiku 5.5 is described as built for high-volume work like classification, extraction and routing.

Compare cost per task, not per token

A cheaper token isn't always a cheaper task. Two things move the real number:

• Tokenizers differ. Claude 4.7 and later models use a newer tokenizer that produces roughly 30% more tokens for the same text. Count tokens with the specific model you plan to use.

• Thinking adds tokens. A model that reasons at length can spend more on one answer than a smaller model spends on three.

The effort dial

Before you switch models, try turning the dial. The effort setting controls how many tokens Claude spends on a response, including text, tool calls and thinking:

response = client.messages.create(
    model="claude-sonnet-5-5",
    max_tokens=1024,
    output_config={"effort": "low"},
    messages=[{"role": "user", "content": prompt}],
)

There are five levels: low, medium, high, xhigh and max. Defaults differ by model. Sonnet 5.5 defaults to high, while Opus 5.5 and Haiku 5.5 default to medium.

Anthropic's advice is to use low and medium liberally wherever your evals show quality holds. Pick one level per workload, because changing it between requests breaks the cache.

The cascade

You often can't tell in advance which requests are easy. A cascade lets the data decide. Send everything to the cheap model first, check the answer, and escalate only when the check fails:

flowchart LR
    T["Ticket"] --> H["Haiku 5.5"]
    H --> V{"Valid answer?"}
    V -->|"yes"| Done["Use it"]
    V -->|"no"| S["Sonnet 5.5"]
    S --> V2{"Valid answer?"}
    V2 -->|"yes"| Done
    V2 -->|"no"| Hu["Human review"]

Here's a version for a ticket classifier. The check is deliberately plain. The answer must be one of four labels:

from dataclasses import dataclass, field
from typing import Callable

LABELS = {"billing", "bug", "feature_request", "other"}


@dataclass
class Tier:
    name: str
    call: Callable[[str], str]


@dataclass
class Outcome:
    answer: str | None
    tier: str | None  # which tier produced the accepted answer
    tried: list[str] = field(default_factory=list)


def is_valid_label(answer: str) -> bool:
    """A cheap, programmatic check. Better than asking the model how sure it is."""
    return answer.strip().lower() in LABELS


def cascade(task: str, tiers: list[Tier], is_good: Callable[[str], bool] = is_valid_label) -> Outcome:
    """Try the cheapest tier first. Move up when its answer fails the check, or when it errors."""
    tried = []
    for tier in tiers:
        tried.append(tier.name)
        try:
            answer = tier.call(task)
        except Exception:
            continue
        if is_good(answer):
            return Outcome(answer.strip().lower(), tier.name, tried)
    return Outcome(None, None, tried)  # nobody gave a usable answer: queue it for a human


def escalation_rate(outcomes: list[Outcome]) -> float:
    """Share of tasks that needed more than the first tier. Watch this number."""
    return sum(len(o.tried) > 1 for o in outcomes) / len(outcomes) if outcomes else 0.0

Wiring it to real models takes a few lines:

import anthropic
from cascade import Tier, cascade

client = anthropic.Anthropic()
SYSTEM = [{
    "type": "text",
    "text": "Classify the ticket as billing, bug, feature_request or other. Reply with the label only.\n\n...your examples...",
    "cache_control": {"type": "ephemeral"},
}]


def ask(model):
    def call(ticket: str) -> str:
        response = client.messages.create(
            model=model,
            max_tokens=10,
            system=SYSTEM,
            messages=[{"role": "user", "content": ticket}],
        )
        return response.content[0].text
    return call


tiers = [Tier("haiku", ask("claude-haiku-5-5")), Tier("sonnet", ask("claude-sonnet-5-5"))]
print(cascade("I was charged twice this month", tiers))

Three things to know before you trust it:

• A valid answer isn't necessarily a correct one. The check above catches malformed output, not wrong labels. A cheap model can confidently pick the wrong valid label. Measure accuracy on labeled examples before you ship.

• Don't ask the model how sure it is. Self-reported confidence is a weak signal. Prefer checks your code can run: a schema, a regex, a test, a lookup.

• Watch the escalation rate. escalation_rate tells you what share of traffic needed the second tier. If it creeps up, your cheap tier is getting worse, or your traffic changed.

Put it together

Here's a worked example. A ticket classifier handles 1,000,000 requests a month. Each one has a 6,000-token stable prefix (instructions and examples), 400 tokens of new input, and 150 tokens of output. All prices are from the table above, and all numbers come from costs.py:

A few lessons hide in that table:

• Caching did most of the work in B. The prefix is 94% of the input, and output is only about 10% of the baseline bill. For this shape of workload, shortening answers would barely matter.

• D barely beats B. Batch halves the price, but I assumed a lower cache hit rate for batch, as the docs suggest. If batch hit rates reached 95%, D would cost $1,810. A batch that misses the cache gives back a lot of its discount.

• E is tempting and dangerous. It's the cheapest row, but only if Haiku's quality holds on your data. That's what the cascade in F protects.

• The cascade is forgiving. In this model it stays cheaper than caching Sonnet alone until about 94% of requests escalate. At a 60% escalation rate it still saves 83%. Your real constraint is accuracy and latency, not price.

One caution on F and G: the escalated requests hit the strong model at a lower volume, so its cache may go cold between requests. If traffic is thin, the longer 1-hour cache keeps that second tier warm.

Smaller wins

Once the big three are in place, these trim the rest:

• Cap max_tokens to what the task needs, since output is the expensive side.

• Ask for compact output. A label or a small JSON object costs far less than a paragraph.

• Trim the prompt. Dead instructions and redundant examples are paid for on every call.

• Fetch instead of stuffing. Retrieve the few relevant passages instead of sending the whole document set.

• Summarize old turns in long conversations instead of replaying them in full.

• Dedupe identical requests with an ordinary response cache for exact repeats. That skips the model call entirely.

• Set a spend limit and alerts, so a retry loop can't run up a bill overnight.

Guardrails

Every one of these levers can hurt quality if you apply it carelessly.

• Build a small labeled test set first. A few hundred real examples is enough to compare setups.

• Track cost per correct answer, not just cost per request. A cheap wrong answer isn't cheap.

• Change one thing at a time, so you know which change moved the number.

• Roll out gradually. Send 5% of traffic through the new path before you flip the rest.

• Watch three numbers: cache hit rate, escalation rate, and the share of work that runs in batch.

• Re-check prices every so often. They change, and so do the cache multipliers.

Other providers

The same three levers exist almost everywhere, with different details:

OpenAI's cached-input rate depends on the model, so check its pricing page for the current figure.

What to try next

Start with measurement. Log request_cost and cache_hit_rate for a week and see where the money goes. Then pull the lever that matches what you find.

• Mostly input, repeated prefix: caching

• Mostly work that can wait: batching

• Mostly easy requests: a smaller model or a cascade

• Mostly long outputs: lower effort, tighter max_tokens, shorter responses

Check the math

The cost numbers in this post come from tests with hand-computed expected values. Create test_costs.py:

import math
from types import SimpleNamespace

from costs import Workload, cache_hit_rate, cascade_cost, monthly_cost, request_cost

SONNET, HAIKU = "claude-sonnet-5-5", "claude-haiku-5-5"
TICKETS = Workload(requests=1_000_000, prefix_tokens=6_000, new_tokens=400, output_tokens=150)


def close(a, b):
    return math.isclose(a, b, rel_tol=1e-9, abs_tol=1e-12)


def usage(input_tokens=0, cache_creation=0, cache_read=0, output=0):
    return SimpleNamespace(input_tokens=input_tokens, cache_creation_input_tokens=cache_creation,
                           cache_read_input_tokens=cache_read, output_tokens=output)


def test_one_request_priced_from_its_usage_fields():
    u = usage(input_tokens=1_000, cache_read=5_000, output=200)
    # 1,000 x $2 + 5,000 x $0.10 + 200 x $10, per million
    assert close(request_cost(SONNET, u), 0.0045)
    assert close(request_cost(SONNET, u, batch=True), 0.00225)


def test_usage_fields_can_be_none_when_caching_is_off():
    u = usage(input_tokens=1_000, output=200)
    u.cache_creation_input_tokens = u.cache_read_input_tokens = None
    assert close(request_cost(SONNET, u), 0.004)       # 1,000 x $2 + 200 x $10, per million
    assert cache_hit_rate(u) == 0.0


def test_cache_hit_rate_from_usage():
    assert close(cache_hit_rate(usage(input_tokens=400, cache_read=6_000)), 6_000 / 6_400)
    assert cache_hit_rate(usage()) == 0.0


def test_baseline_month():
    assert close(monthly_cost(SONNET, TICKETS), 14_300)      # 12,800 input + 1,500 output


def test_caching_at_95_percent_hits():
    assert close(monthly_cost(SONNET, TICKETS, cache_hit_share=0.95), 3_620)


def test_batch_alone_halves_the_bill():
    assert close(monthly_cost(SONNET, TICKETS, batch=True), 7_150)


def test_cache_with_no_hits_costs_more_than_no_cache():
    assert monthly_cost(SONNET, TICKETS, cache_hit_share=0.0) > monthly_cost(SONNET, TICKETS)


def test_cache_break_even_is_about_one_hit_in_five():
    no_cache = monthly_cost(SONNET, TICKETS)
    assert monthly_cost(SONNET, TICKETS, cache_hit_share=0.25) < no_cache
    assert monthly_cost(SONNET, TICKETS, cache_hit_share=0.20) > no_cache


def test_smaller_model_is_far_cheaper_for_the_same_work():
    assert close(monthly_cost(HAIKU, TICKETS), 715)
    assert monthly_cost(HAIKU, TICKETS) < monthly_cost(SONNET, TICKETS) / 15


def test_cascade_edges():
    assert close(cascade_cost(HAIKU, SONNET, TICKETS, 0.0), monthly_cost(HAIKU, TICKETS))
    both = monthly_cost(HAIKU, TICKETS) + monthly_cost(SONNET, TICKETS)
    assert close(cascade_cost(HAIKU, SONNET, TICKETS, 1.0), both)


def test_cascade_with_caching():
    # Haiku sees everything ($209.50), Sonnet sees the 30% that failed validation ($1,086)
    assert close(cascade_cost(HAIKU, SONNET, TICKETS, 0.30, cache_hit_share=0.95), 1_295.5)

And test_cascade.py:

from cascade import Tier, cascade, escalation_rate


def make_tier(name, answer, calls):
    def call(task):
        calls.append(name)
        if isinstance(answer, Exception):
            raise answer
        return answer
    return Tier(name, call)


def test_cheap_tier_wins_when_its_answer_is_valid():
    calls = []
    out = cascade("my invoice is wrong", [make_tier("haiku", " Billing ", calls), make_tier("sonnet", "bug", calls)])
    assert (out.answer, out.tier, out.tried) == ("billing", "haiku", ["haiku"])
    assert calls == ["haiku"]                         # the expensive model was never called


def test_invalid_answer_moves_up_a_tier():
    calls = []
    out = cascade("???", [make_tier("haiku", "I think it might be a billing issue", calls), make_tier("sonnet", "billing", calls)])
    assert (out.answer, out.tier, out.tried) == ("billing", "sonnet", ["haiku", "sonnet"])


def test_an_error_moves_up_a_tier():
    calls = []
    out = cascade("???", [make_tier("haiku", TimeoutError("slow"), calls), make_tier("sonnet", "bug", calls)])
    assert out.tier == "sonnet" and out.answer == "bug"


def test_nobody_gives_a_usable_answer():
    calls = []
    out = cascade("???", [make_tier("haiku", "dunno", calls), make_tier("sonnet", "hmm", calls)])
    assert out.answer is None and out.tier is None and out.tried == ["haiku", "sonnet"]


def test_escalation_rate():
    haiku_right = lambda task: "billing" if "invoice" in task else "not sure"
    tiers = [Tier("haiku", haiku_right), Tier("sonnet", lambda task: "other")]
    tasks = ["invoice 1", "invoice 2", "invoice 3", "weird thing"]
    outcomes = [cascade(t, tiers) for t in tasks]
    assert escalation_rate(outcomes) == 0.25
    assert escalation_rate([]) == 0.0

Run both with pytest -q. No fixtures or plugins are needed.

Recap

• Tokens are the bill, and output tokens cost the most.

• Caching: put the stable part first, put the breakpoint at its end, and check the usage fields.

• Batching: half price for anything nobody is waiting on.

• Smaller models: big savings, so protect quality with evals and a cascade.

• Combine them, but measure each step.

Prices and limits checked against Anthropic's, OpenAI's and Google's documentation on October 29, 2026. They change often, so confirm them on the pricing pages before you plan a budget.

Sources

• Anthropic: Pricing

• Anthropic: Prompt caching

• Anthropic: Batch processing

• Anthropic: Effort

• OpenAI: Prompt caching

• OpenAI: Batch API

• Google: Gemini API optimization and inference

• Google: Gemini API context caching