--- title: "How Do I Know When a Third-Party API Quietly Changes Its Response?" author: "Anshu Pathak" published: "2026-11-15T05:17:02.324Z" updated: "2026-11-15T05:17:02.324Z" url: "https://zyvop.com/how-do-i-know-when-a-third-party-api-quietly-changes-its-response-9lr1v" canonical: "https://zyvop.com/how-do-i-know-when-a-third-party-api-quietly-changes-its-response-9lr1v" tags: ["monitoring","backend","Python","automation","api"] --- # How Do I Know When a Third-Party API Quietly Changes Its Response? A daily script that snapshots the shape of an API's JSON, diffs it, and warns you before your app breaks. ![Cover image]() Your app worked yesterday. Today a customer reports a blank page, and an hour of digging later you find the cause: a field that used to be a string is now a number, or a key you depend on is gone. Nobody emailed you. The status page is green. The changelog says nothing. Third-party APIs change quietly all the time, and the damage shows up far away from the cause. This post answers the questions I had when I wanted to catch it early, and ends with a dependency-free Python script you can run from cron. ## What does "quietly changes" actually look like? Rarely a dramatic breaking change. Usually one of these: - A field is renamed or disappears. - A number becomes a string, or the reverse (`"12.50"` becomes `12.5`). - A field that was always present can now be `null`. - An object becomes a list, or gets wrapped in `{ "data": ... }`. - A new field appears. Harmless today, but often the first sign of a bigger rollout. - A `200 OK` response starts carrying an error body. Every one of these returns HTTP 200, so any check that only looks at status codes stays green. ## Why don't my tests and uptime monitor catch this? An uptime monitor checks that the endpoint answers, not what it says. Your tests usually run against fixtures recorded months ago, so they keep passing against an API that no longer behaves that way. Contract-testing tools like Pact work well when you and the provider cooperate, but you can't make a third party sign a contract with you. The only source of truth is the live response, so you have to look at it regularly. ## Can't I just diff the response? Not directly. Values change on every call: timestamps, prices, counters. A plain diff would alert you every single day. The trick is to diff the **shape** instead. Reduce the response to a map from each JSON path to the set of types seen there, and throw the values away. This response: ```json {"id": 7, "price": "12.50", "tags": ["a", "b"], "owner": {"name": "Asha"}} ``` becomes: ```text $ object $.id number $.price string $.tags array $.tags[] string $.owner object $.owner.name string ``` Tomorrow `id` can be 8 and `price` can be `"13.00"`, and nothing changes. But if `price` becomes `13.0` (a number), the line `$.price string` turns into `$.price number`, and that is drift. Arrays are merged: every item in a list feeds the same `[]` path, so a list of 1,000 consistent items costs the same as a list of one. ## How does the whole thing fit together? ```mermaid flowchart LR A[Cron runs daily] --> B[Fetch each endpoint] B --> C[Save raw sample] B --> D[Reduce JSON to its shape] D --> E{Same as baseline?} E -- yes --> F[Stay quiet] E -- no --> G[Send alert] G --> H[Update baseline] ``` The baseline is just a JSON file next to the script. The first run records it, and every later run compares against it. ## What's the script? It uses only the standard library, so any modern Python 3 (3.6 or newer) will run it. Save it as `api_drift.py`. ```python #!/usr/bin/env python3 # api_drift.py - warn me when a third-party API changes the shape of its JSON. import json import os import sys import urllib.request from pathlib import Path STATE = Path('api_shapes.json') # the shapes we remember SAMPLES = Path('samples') # latest raw response per endpoint, for debugging GRACE = 3 # runs a field may be absent before it counts as removed ENDPOINTS = [ {'name': 'github-cpython', 'url': 'https://api.github.com/repos/python/cpython'}, {'name': 'vendor-rates', # replace with your own 'url': 'https://api.example.com/v1/rates', 'headers': {'Authorization': 'Bearer ${VENDOR_TOKEN}'}, 'ignore': ['$.rates']}, # a map keyed by currency code ] def type_of(v): if v is None: return 'null' if isinstance(v, bool): # check bool first: True is an int in Python return 'boolean' if isinstance(v, (int, float)): return 'number' if isinstance(v, str): return 'string' return 'array' if isinstance(v, list) else 'object' def shape(value, ignore, path='$', out=None): # Reduce any JSON value to {path: {types}} - the values are thrown away. out = {} if out is None else out out.setdefault(path, set()).add(type_of(value)) if path in ignore: # dynamic keys: record the node, skip its children return out if isinstance(value, dict): for key, child in value.items(): shape(child, ignore, f'{path}.{key}', out) elif isinstance(value, list): for child in value: shape(child, ignore, f'{path}[]', out) return out def fmt(types): return '/'.join(sorted(types)) def fetch(url, headers): headers = {k: os.path.expandvars(v) for k, v in headers.items()} req = urllib.request.Request(url, headers={'User-Agent': 'api-drift/1.0', **headers}) with urllib.request.urlopen(req, timeout=20) as resp: return json.load(resp) def check(ep, state): name = ep['name'] try: data = fetch(ep['url'], ep.get('headers', {})) except Exception as err: return [f'{name}: request failed ({err})'] SAMPLES.mkdir(exist_ok=True) (SAMPLES / f'{name}.json').write_text(json.dumps(data, indent=2)) current = shape(data, ep.get('ignore', [])) entry = state.get(name) if entry is None: # first run: just learn the shape state[name] = {'shape': {p: sorted(t) for p, t in current.items()}, 'missing': {}} print(f'{name}: baseline saved ({len(current)} paths)') return [] known = {p: set(t) for p, t in entry['shape'].items()} missing = entry['missing'] problems = [] for path, types in current.items(): if path not in known: problems.append(f'{name}: NEW {path} ({fmt(types)})') known[path] = set(types) elif types - known[path]: problems.append(f'{name}: TYPE {path} was {fmt(known[path])}, ' f'now also {fmt(types - known[path])}') known[path] |= types missing.pop(path, None) for path in [p for p in known if p not in current]: missing[path] = missing.get(path, 0) + 1 if missing[path] >= GRACE: problems.append(f'{name}: REMOVED {path} (gone for {GRACE} runs)') del known[path] del missing[path] entry['shape'] = {p: sorted(t) for p, t in known.items()} return problems def alert(problems): text = 'API drift detected:\n' + '\n'.join(problems) print(text) hook = os.environ.get('ALERT_WEBHOOK') # Slack-style webhook; Discord wants 'content' if hook: req = urllib.request.Request(hook, json.dumps({'text': text}).encode(), {'Content-Type': 'application/json'}) urllib.request.urlopen(req, timeout=10) def save(state): tmp = STATE.with_suffix('.tmp') tmp.write_text(json.dumps(state, indent=2, sort_keys=True)) os.replace(tmp, STATE) # atomic: a crash can't leave half a file def main(): state = json.loads(STATE.read_text()) if STATE.exists() else {} problems = [] for ep in ENDPOINTS: problems += check(ep, state) if problems: alert(problems) # if this raises, state isn't saved, so we alert again next run save(state) sys.exit(1 if problems else 0) if __name__ == '__main__': main() ``` ## How do I set it up? 1. Put your endpoints in `ENDPOINTS`. Use `${VAR}` in headers so tokens stay in the environment and out of the file. 2. Run it once by hand. The first run records a baseline for each endpoint and prints `baseline saved`. 3. Schedule it. Put the secrets at the top of your crontab, then add one line: ```text ALERT_WEBHOOK=https://hooks.slack.com/services/XXX VENDOR_TOKEN=your-token-here 0 7 * * * cd /home/me/api-drift && /usr/bin/python3 api_drift.py >> drift.log 2>&1 ``` Cron uses the server's timezone, so check what 7:00 means on your machine. The script also exits with code 1 when it finds drift, so any scheduler that watches exit codes can pick it up. ## What does an alert look like? ```text API drift detected: vendor-rates: TYPE $.items[].price was string, now also number vendor-rates: NEW $.items[].currency (string) vendor-rates: REMOVED $.items[].legacy_id (gone for 3 runs) ``` Each line names the endpoint, the kind of change, and the exact JSON path, so you know where to look. ## How does it avoid crying wolf? Four choices keep the noise down: - **Types, not values.** Prices and timestamps can change all day without a single alert. - **The baseline only widens.** If a field is sometimes `null`, you get one alert the first time it happens. After that, `null` is part of the known shape and it stops nagging. - **Removal needs three runs in a row.** A field missing from one sample isn't proof it's gone, so `GRACE` waits before calling it removed. - **Dynamic keys can be ignored.** A map keyed by ID, date, or currency code would create a new path for every key. List that path in `ignore` and its children are skipped. One more detail: the alert goes out before the state file is saved. If the webhook fails, the script crashes without saving, so tomorrow's run reports the same drift again instead of swallowing it. ## What won't this catch? - **Changes in meaning.** If prices move from rupees to paise, they are still numbers. Units, sort order, and ID formats are invisible by design. - **Paths your sample never touches.** If you call `/orders?limit=5`, you won't see what `/orders?status=refunded` returns. Add an entry to `ENDPOINTS` for each variation you depend on. - **Rarely present fields.** A field that appears in one response out of a thousand may not show up in your sample for weeks. When it finally does, you get a single NEW alert. - **Behaviour.** Rate limits, pagination rules, and auth changes mostly show up as failed requests, not as shape changes. Also, the script is silent when nothing changes, so you can't tell "no drift" from "cron stopped running". Pair it with a dead-man's-switch ping so a missing run raises its own alarm. ## When I get an alert, what do I do? 1. Open `samples/.json`. It holds the latest raw response from that endpoint. 2. Search your code for the path that changed and see how each place uses it. 3. Check the provider's changelog and docs. If there is nothing, send them the sample in a support ticket. 4. Validate responses at the edge of your code (with pydantic or jsonschema, for example) so the next surprise fails loudly in one place instead of as a blank page three layers away. ## Why not use an OpenAPI diff tool instead? If the provider publishes an OpenAPI spec, a tool like oasdiff can compare versions of it, and that is worth doing. But many APIs publish no spec, and a spec describes what the provider intends, not what the server actually returns. This script checks real responses, so it works with any JSON API and catches the cases where the docs and reality disagree. ## The takeaway You can't stop a third party from changing its API, but you can find out the next morning instead of from a customer. Diff the shape, ignore the values, alert once per change, and keep the raw sample so you can see exactly what moved.