--- title: "Where Did the LLM Tokens Go? A 100-Line Proxy That Finds Your Priciest Prompt" author: "Anshu Pathak" published: "2026-11-07T07:39:19.599Z" updated: "2026-11-07T07:40:05.166Z" url: "https://zyvop.com/where-did-the-llm-tokens-go-a-100-line-proxy-that-finds-your-priciest-prompt-b1pxy" canonical: "https://zyvop.com/where-did-the-llm-tokens-go-a-100-line-proxy-that-finds-your-priciest-prompt-b1pxy" tags: ["Proxy","Python","cost","SQLite","LLM"] --- # Where Did the LLM Tokens Go? A 100-Line Proxy That Finds Your Priciest Prompt A small Python proxy that logs LLM tokens per feature and prompt, shows what each one costs, and fails over to a backup provider. ![Cover image]() Your LLM bill says $144 this week. Which feature spent it? Providers typically break usage down by model, API key or project, not by the part of your app that made the call. So unless every feature has its own key, you guess, and guessing is how you spend a day shrinking a prompt that was 2% of the bill. The fix is a small proxy between your app and the provider. It tags each call with the feature that made it, adds up the tokens, and lets one query show who is spending what. ```mermaid flowchart LR A[Your app] -->|X-Feature header| P[tokenlens proxy] P -->|primary| U1[LLM provider] P -.->|on 429 or 5xx| U2[Fallback provider] P --> D[(SQLite tokens per call)] D --> R[Report cost and share per prompt] ``` ## How it works - Your app talks to the proxy instead of the provider, with one extra header per call: `X-Feature: doc-summary`. - The proxy forwards the request, reads the `usage` numbers from the response, and writes one row to SQLite. It never stores prompt text, only a short hash of the system prompt, so two versions of a prompt show up as two separate rows. If your system prompt embeds per-request data (the document itself, say), that hash changes on every call. Send an `X-Prompt: doc-summary-v1` header instead and the proxy uses that name. - If the provider answers 429 or a 5xx error, or can't be reached, the proxy retries on a fallback provider before your app sees anything. ## The script Standard library only, Python 3.9+. Save it as `tokenlens.py`: ```python #!/usr/bin/env python3 """tokenlens: a logging proxy for OpenAI-compatible APIs. python tokenlens.py # proxy on 127.0.0.1:8787 python tokenlens.py report 7 # who spent what, last 7 days Headers: X-Feature = which part of your app; X-Prompt = optional prompt name """ import hashlib, http.client, json, os, sqlite3, sys, time, urllib.parse from http.server import BaseHTTPRequestHandler, ThreadingHTTPServer DB = os.environ.get("TOKENLENS_DB", "tokenlens.db") UPSTREAMS = [ # tried in order; "model" overrides the model name (None = keep it) {"base": os.environ.get("PRIMARY_BASE", "https://api.openai.com/v1"), "key": os.environ.get("PRIMARY_KEY", ""), "model": None}] if os.environ.get("FALLBACK_BASE"): UPSTREAMS.append({"base": os.environ["FALLBACK_BASE"], "key": os.environ.get("FALLBACK_KEY", ""), "model": os.environ.get("FALLBACK_MODEL")}) # USD per 1M tokens (input, output). Placeholders: use your provider's current prices. PRICES = {"model-a": (0.50, 1.50), "model-b": (3.00, 15.00)} def db(): c = sqlite3.connect(DB, timeout=10) c.execute("PRAGMA journal_mode=WAL") c.execute("CREATE TABLE IF NOT EXISTS calls(ts REAL, feature TEXT, prompt TEXT, model TEXT," " tin INT, tout INT, ms INT, status INT, upstream INT)") return c class Proxy(BaseHTTPRequestHandler): def log_message(self, *args): pass def do_POST(self): body = json.loads(self.rfile.read(int(self.headers.get("Content-Length", 0))) or b"{}") feature = self.headers.get("X-Feature", "untagged") system = next((m.get("content") for m in body.get("messages") or [] if m.get("role") in ("system", "developer")), None) # hashed, never stored prompt = self.headers.get("X-Prompt") or hashlib.sha1(json.dumps(system).encode()).hexdigest()[:8] stream = bool(body.get("stream")) if stream and self.path.endswith("/chat/completions"): # ask for the usage chunk, or streams log 0 tokens body["stream_options"] = {**(body.get("stream_options") or {}), "include_usage": True} t0, usage, status, model, used = time.time(), {}, 502, body.get("model"), -1 for i, up in enumerate(UPSTREAMS): sent = dict(body, model=up["model"] or body.get("model")) u = urllib.parse.urlparse(up["base"]) Conn = http.client.HTTPSConnection if u.scheme == "https" else http.client.HTTPConnection try: conn = Conn(u.netloc, timeout=120) conn.request("POST", u.path + self.path.removeprefix("/v1"), json.dumps(sent), {"Content-Type": "application/json", "Authorization": "Bearer " + up["key"]}) r = conn.getresponse() except OSError: continue if (r.status == 429 or r.status >= 500) and i < len(UPSTREAMS) - 1: r.read(); conn.close(); continue # fail over before sending a byte to the client status, model, used = r.status, sent["model"], i self.send_response(status) self.send_header("Content-Type", r.getheader("Content-Type", "application/json")) try: if stream: self.end_headers() for line in iter(r.readline, b""): self.wfile.write(line); self.wfile.flush() d = line[5:].strip() if line.startswith(b"data:") else b"" if d.startswith(b"{"): try: # chat: top-level usage; Responses API: inside "response" ev = json.loads(d) usage = ev.get("usage") or (ev.get("response") or {}).get("usage") or usage except (ValueError, AttributeError): pass else: data = r.read() self.send_header("Content-Length", str(len(data))) self.end_headers(); self.wfile.write(data) try: usage = json.loads(data).get("usage") or {} except (ValueError, AttributeError): pass except (OSError, http.client.HTTPException): pass # client or upstream dropped mid-response; still log the call finally: conn.close() break else: self.send_error(502, "all upstreams failed") tin = usage.get("prompt_tokens") or usage.get("input_tokens") or 0 tout = usage.get("completion_tokens") or usage.get("output_tokens") or 0 c = db() c.execute("INSERT INTO calls VALUES(?,?,?,?,?,?,?,?,?)", (t0, feature, prompt, model, tin, tout, int((time.time() - t0) * 1000), status, used)) c.commit(); c.close() def report(days=7): rows = db().execute("SELECT feature, prompt, model, COUNT(*), SUM(tin), SUM(tout) FROM calls " "WHERE ts > ? AND status < 400 GROUP BY 1, 2, 3", (time.time() - days * 86400,)).fetchall() cost = lambda m, i, o: (i * PRICES.get(m, (0, 0))[0] + o * PRICES.get(m, (0, 0))[1]) / 1e6 table = sorted(((cost(m, i, o), f, p, m, n, i, o) for f, p, m, n, i, o in rows), reverse=True) total = sum(r[0] for r in table) or 1 print(f"{'feature':<18}{'prompt':<14}{'model':<9}{'calls':>6}{'in':>10}{'out':>9}{'cost $':>9}{'share':>7}") for c, f, p, m, n, i, o in table: print(f"{f:<18}{p:<14}{m:<9}{n:>6}{i:>10}{o:>9}{c:>9.2f}{c / total:>7.0%}") if __name__ == "__main__": if sys.argv[1:2] == ["report"]: report(int(sys.argv[2]) if len(sys.argv) > 2 else 7) else: db().close() print("tokenlens listening on http://127.0.0.1:8787") ThreadingHTTPServer(("127.0.0.1", 8787), Proxy).serve_forever() ``` Start it, keeping your provider key here and out of the app: ```bash export PRIMARY_KEY=sk-your-key # PRIMARY_BASE defaults to https://api.openai.com/v1 python tokenlens.py # listens on 127.0.0.1:8787 ``` For failover, also set `FALLBACK_BASE`, `FALLBACK_KEY` and `FALLBACK_MODEL` (the fallback's name for the model). Then point your app at the proxy and tag each feature. With the OpenAI Python SDK: ```python from openai import OpenAI client = OpenAI(base_url="http://127.0.0.1:8787/v1", api_key="unused", default_headers={"X-Feature": "doc-summary"}) ``` Use `extra_headers={"X-Feature": "email-reply"}` on a single call to override the default. Last, edit `PRICES` so the report uses your models' current prices. ## Reading the report After a few days of traffic: ```bash python tokenlens.py report 7 ``` Example output, from a simulated week with placeholder prices. Yours will differ: ```text feature prompt model calls in out cost $ share doc-summary 9f3a1c02 model-b 1700 22000000 1300000 85.50 60% email-reply c71d0e55 model-b 1200 3600000 900000 24.30 17% search-rerank 4be8a9d1 model-a 18000 27000000 1800000 16.20 11% support-chat 2a6f77b3 model-a 5000 15000000 4000000 13.50 9% tagging e03b5c48 model-a 9500 4700000 200000 2.65 2% untagged 5d90ae17 model-a 900 2000000 300000 1.45 1% ``` One row is 60% of the spend. `doc-summary` sends about 13,000 input tokens per call (22M tokens over 1,700 calls), most likely because the whole document goes in with every request, and it runs on the expensive model. `search-rerank` makes more than ten times as many calls and costs 11%. Call counts mislead. Tokens times price doesn't. Fix the top row first. These are the usual levers, roughly in order of payoff: 1. **Send less.** Pass the relevant chunks instead of the whole document. 2. **Route easy cases to the cheaper model.** Keep the expensive one for the hard ones. 3. **Use prompt caching** for the repeated prefix, if your provider supports it. 4. **Cap** `max_tokens` on replies that never need to be long. Run the report again a week later. The row should shrink, and now you can prove it. ## Things to know - **It has no auth and holds your provider keys.** It binds to `127.0.0.1`. Keep it on localhost or inside a private network. - **Streaming needs one extra field.** For Chat Completions streams, the proxy adds `stream_options: {"include_usage": true}`, the OpenAI-compatible way to get token counts at the end of a stream. If your provider rejects that field, remove that line, but streamed calls will then log zero tokens. - **Costs are estimates.** The report applies one input price and one output price per model, so cached-input discounts and other special rates aren't reflected. Calls cut off before the final usage chunk (a dropped stream, a hung-up client) are logged with zero tokens, so they are under-counted. Compare the totals with your invoice. - **Scope.** It proxies OpenAI-compatible `POST` endpoints and reads usage from Chat Completions replies and from Responses-style replies, where a stream carries the usage in its final event. A fallback provider must speak the same format. Add one header per call site, wait a week, and optimise whichever row is at the top.