Where Did the LLM Tokens Go? A 100-Line Proxy That Finds Your Priciest Prompt

Your LLM bill says $144 this week. Which feature spent it? Providers typically break usage down by model, API key or project, not by the part of your app that made the call. So unless every feature has its own key, you guess, and guessing is how you spend a day shrinking a prompt that was 2% of the bill.

The fix is a small proxy between your app and the provider. It tags each call with the feature that made it, adds up the tokens, and lets one query show who is spending what.

flowchart LR
    A[Your app] -->|X-Feature header| P[tokenlens proxy]
    P -->|primary| U1[LLM provider]
    P -.->|on 429 or 5xx| U2[Fallback provider]
    P --> D[(SQLite tokens per call)]
    D --> R[Report cost and share per prompt]

How it works

• Your app talks to the proxy instead of the provider, with one extra header per call: X-Feature: doc-summary.

• The proxy forwards the request, reads the usage numbers from the response, and writes one row to SQLite. It never stores prompt text, only a short hash of the system prompt, so two versions of a prompt show up as two separate rows. If your system prompt embeds per-request data (the document itself, say), that hash changes on every call. Send an X-Prompt: doc-summary-v1 header instead and the proxy uses that name.

• If the provider answers 429 or a 5xx error, or can't be reached, the proxy retries on a fallback provider before your app sees anything.

The script

Standard library only, Python 3.9+. Save it as tokenlens.py:

#!/usr/bin/env python3
"""tokenlens: a logging proxy for OpenAI-compatible APIs.
   python tokenlens.py            # proxy on 127.0.0.1:8787
   python tokenlens.py report 7   # who spent what, last 7 days
   Headers: X-Feature = which part of your app; X-Prompt = optional prompt name
"""
import hashlib, http.client, json, os, sqlite3, sys, time, urllib.parse
from http.server import BaseHTTPRequestHandler, ThreadingHTTPServer

DB = os.environ.get("TOKENLENS_DB", "tokenlens.db")
UPSTREAMS = [  # tried in order; "model" overrides the model name (None = keep it)
    {"base": os.environ.get("PRIMARY_BASE", "https://api.openai.com/v1"),
     "key": os.environ.get("PRIMARY_KEY", ""), "model": None}]
if os.environ.get("FALLBACK_BASE"):
    UPSTREAMS.append({"base": os.environ["FALLBACK_BASE"],
                      "key": os.environ.get("FALLBACK_KEY", ""),
                      "model": os.environ.get("FALLBACK_MODEL")})
# USD per 1M tokens (input, output). Placeholders: use your provider's current prices.
PRICES = {"model-a": (0.50, 1.50), "model-b": (3.00, 15.00)}

def db():
    c = sqlite3.connect(DB, timeout=10)
    c.execute("PRAGMA journal_mode=WAL")
    c.execute("CREATE TABLE IF NOT EXISTS calls(ts REAL, feature TEXT, prompt TEXT, model TEXT,"
              " tin INT, tout INT, ms INT, status INT, upstream INT)")
    return c

class Proxy(BaseHTTPRequestHandler):
    def log_message(self, *args):
        pass

    def do_POST(self):
        body = json.loads(self.rfile.read(int(self.headers.get("Content-Length", 0))) or b"{}")
        feature = self.headers.get("X-Feature", "untagged")
        system = next((m.get("content") for m in body.get("messages") or []
                       if m.get("role") in ("system", "developer")), None)  # hashed, never stored
        prompt = self.headers.get("X-Prompt") or hashlib.sha1(json.dumps(system).encode()).hexdigest()[:8]
        stream = bool(body.get("stream"))
        if stream and self.path.endswith("/chat/completions"):  # ask for the usage chunk, or streams log 0 tokens
            body["stream_options"] = {**(body.get("stream_options") or {}), "include_usage": True}
        t0, usage, status, model, used = time.time(), {}, 502, body.get("model"), -1
        for i, up in enumerate(UPSTREAMS):
            sent = dict(body, model=up["model"] or body.get("model"))
            u = urllib.parse.urlparse(up["base"])
            Conn = http.client.HTTPSConnection if u.scheme == "https" else http.client.HTTPConnection
            try:
                conn = Conn(u.netloc, timeout=120)
                conn.request("POST", u.path + self.path.removeprefix("/v1"), json.dumps(sent),
                             {"Content-Type": "application/json", "Authorization": "Bearer " + up["key"]})
                r = conn.getresponse()
            except OSError:
                continue
            if (r.status == 429 or r.status >= 500) and i < len(UPSTREAMS) - 1:
                r.read(); conn.close(); continue  # fail over before sending a byte to the client
            status, model, used = r.status, sent["model"], i
            self.send_response(status)
            self.send_header("Content-Type", r.getheader("Content-Type", "application/json"))
            try:
                if stream:
                    self.end_headers()
                    for line in iter(r.readline, b""):
                        self.wfile.write(line); self.wfile.flush()
                        d = line[5:].strip() if line.startswith(b"data:") else b""
                        if d.startswith(b"{"):
                            try:  # chat: top-level usage; Responses API: inside "response"
                                ev = json.loads(d)
                                usage = ev.get("usage") or (ev.get("response") or {}).get("usage") or usage
                            except (ValueError, AttributeError): pass
                else:
                    data = r.read()
                    self.send_header("Content-Length", str(len(data)))
                    self.end_headers(); self.wfile.write(data)
                    try: usage = json.loads(data).get("usage") or {}
                    except (ValueError, AttributeError): pass
            except (OSError, http.client.HTTPException):
                pass  # client or upstream dropped mid-response; still log the call
            finally:
                conn.close()
            break
        else:
            self.send_error(502, "all upstreams failed")
        tin = usage.get("prompt_tokens") or usage.get("input_tokens") or 0
        tout = usage.get("completion_tokens") or usage.get("output_tokens") or 0
        c = db()
        c.execute("INSERT INTO calls VALUES(?,?,?,?,?,?,?,?,?)", (t0, feature, prompt, model, tin,
                  tout, int((time.time() - t0) * 1000), status, used))
        c.commit(); c.close()

def report(days=7):
    rows = db().execute("SELECT feature, prompt, model, COUNT(*), SUM(tin), SUM(tout) FROM calls "
                        "WHERE ts > ? AND status < 400 GROUP BY 1, 2, 3",
                        (time.time() - days * 86400,)).fetchall()
    cost = lambda m, i, o: (i * PRICES.get(m, (0, 0))[0] + o * PRICES.get(m, (0, 0))[1]) / 1e6
    table = sorted(((cost(m, i, o), f, p, m, n, i, o) for f, p, m, n, i, o in rows), reverse=True)
    total = sum(r[0] for r in table) or 1
    print(f"{'feature':<18}{'prompt':<14}{'model':<9}{'calls':>6}{'in':>10}{'out':>9}{'cost $':>9}{'share':>7}")
    for c, f, p, m, n, i, o in table:
        print(f"{f:<18}{p:<14}{m:<9}{n:>6}{i:>10}{o:>9}{c:>9.2f}{c / total:>7.0%}")

if __name__ == "__main__":
    if sys.argv[1:2] == ["report"]:
        report(int(sys.argv[2]) if len(sys.argv) > 2 else 7)
    else:
        db().close()
        print("tokenlens listening on http://127.0.0.1:8787")
        ThreadingHTTPServer(("127.0.0.1", 8787), Proxy).serve_forever()

Start it, keeping your provider key here and out of the app:

export PRIMARY_KEY=sk-your-key        # PRIMARY_BASE defaults to https://api.openai.com/v1
python tokenlens.py                   # listens on 127.0.0.1:8787

For failover, also set FALLBACK_BASE, FALLBACK_KEY and FALLBACK_MODEL (the fallback's name for the model). Then point your app at the proxy and tag each feature. With the OpenAI Python SDK:

from openai import OpenAI

client = OpenAI(base_url="http://127.0.0.1:8787/v1", api_key="unused",
                default_headers={"X-Feature": "doc-summary"})

Use extra_headers={"X-Feature": "email-reply"} on a single call to override the default. Last, edit PRICES so the report uses your models' current prices.

Reading the report

After a few days of traffic:

python tokenlens.py report 7

Example output, from a simulated week with placeholder prices. Yours will differ:

feature           prompt        model     calls        in      out   cost $  share
doc-summary       9f3a1c02      model-b    1700  22000000  1300000    85.50    60%
email-reply       c71d0e55      model-b    1200   3600000   900000    24.30    17%
search-rerank     4be8a9d1      model-a   18000  27000000  1800000    16.20    11%
support-chat      2a6f77b3      model-a    5000  15000000  4000000    13.50     9%
tagging           e03b5c48      model-a    9500   4700000   200000     2.65     2%
untagged          5d90ae17      model-a     900   2000000   300000     1.45     1%

One row is 60% of the spend. doc-summary sends about 13,000 input tokens per call (22M tokens over 1,700 calls), most likely because the whole document goes in with every request, and it runs on the expensive model. search-rerank makes more than ten times as many calls and costs 11%. Call counts mislead. Tokens times price doesn't.

Fix the top row first. These are the usual levers, roughly in order of payoff:

1. Send less. Pass the relevant chunks instead of the whole document.

2. Route easy cases to the cheaper model. Keep the expensive one for the hard ones.

3. Use prompt caching for the repeated prefix, if your provider supports it.

4. Cap max_tokens on replies that never need to be long.

Run the report again a week later. The row should shrink, and now you can prove it.

Things to know

• It has no auth and holds your provider keys. It binds to 127.0.0.1. Keep it on localhost or inside a private network.

• Streaming needs one extra field. For Chat Completions streams, the proxy adds stream_options: {"include_usage": true}, the OpenAI-compatible way to get token counts at the end of a stream. If your provider rejects that field, remove that line, but streamed calls will then log zero tokens.

• Costs are estimates. The report applies one input price and one output price per model, so cached-input discounts and other special rates aren't reflected. Calls cut off before the final usage chunk (a dropped stream, a hung-up client) are logged with zero tokens, so they are under-counted. Compare the totals with your invoice.

• Scope. It proxies OpenAI-compatible POST endpoints and reads usage from Chat Completions replies and from Responses-style replies, where a stream carries the usage in its final event. A fallback provider must speak the same format.

Add one header per call site, wait a week, and optimise whichever row is at the top.