DevOps

How Do I Know When a Third-Party API Quietly Changes Its Response?

A daily script that snapshots the shape of an API's JSON, diffs it, and warns you before your app breaks.

·
7 min read
·
AI-assisted

AI tooling disclosure

Generative AI materially contributed to this article.

Human review confirmed by the author.

How Do I Know When a Third-Party API Quietly Changes Its Response?

Your app worked yesterday. Today a customer reports a blank page, and an hour of digging later you find the cause: a field that used to be a string is now a number, or a key you depend on is gone. Nobody emailed you. The status page is green. The changelog says nothing.

Third-party APIs change quietly all the time, and the damage shows up far away from the cause. This post answers the questions I had when I wanted to catch it early, and ends with a dependency-free Python script you can run from cron.

What does "quietly changes" actually look like?

Rarely a dramatic breaking change. Usually one of these:

  • A field is renamed or disappears.

  • A number becomes a string, or the reverse ("12.50" becomes 12.5).

  • A field that was always present can now be null.

  • An object becomes a list, or gets wrapped in { "data": ... }.

  • A new field appears. Harmless today, but often the first sign of a bigger rollout.

  • A 200 OK response starts carrying an error body.

Every one of these returns HTTP 200, so any check that only looks at status codes stays green.

Why don't my tests and uptime monitor catch this?

An uptime monitor checks that the endpoint answers, not what it says. Your tests usually run against fixtures recorded months ago, so they keep passing against an API that no longer behaves that way. Contract-testing tools like Pact work well when you and the provider cooperate, but you can't make a third party sign a contract with you.

The only source of truth is the live response, so you have to look at it regularly.

Can't I just diff the response?

Not directly. Values change on every call: timestamps, prices, counters. A plain diff would alert you every single day.

The trick is to diff the shape instead. Reduce the response to a map from each JSON path to the set of types seen there, and throw the values away. This response:

{"id": 7, "price": "12.50", "tags": ["a", "b"], "owner": {"name": "Asha"}}

becomes:

$              object
$.id           number
$.price        string
$.tags         array
$.tags[]       string
$.owner        object
$.owner.name   string

Tomorrow id can be 8 and price can be "13.00", and nothing changes. But if price becomes 13.0 (a number), the line $.price string turns into $.price number, and that is drift. Arrays are merged: every item in a list feeds the same [] path, so a list of 1,000 consistent items costs the same as a list of one.

How does the whole thing fit together?

flowchart LR
    A[Cron runs daily] --> B[Fetch each endpoint]
    B --> C[Save raw sample]
    B --> D[Reduce JSON to its shape]
    D --> E{Same as baseline?}
    E -- yes --> F[Stay quiet]
    E -- no --> G[Send alert]
    G --> H[Update baseline]

The baseline is just a JSON file next to the script. The first run records it, and every later run compares against it.

What's the script?

It uses only the standard library, so any modern Python 3 (3.6 or newer) will run it. Save it as api_drift.py.

#!/usr/bin/env python3
# api_drift.py - warn me when a third-party API changes the shape of its JSON.
import json
import os
import sys
import urllib.request
from pathlib import Path

STATE = Path('api_shapes.json')   # the shapes we remember
SAMPLES = Path('samples')         # latest raw response per endpoint, for debugging
GRACE = 3                         # runs a field may be absent before it counts as removed

ENDPOINTS = [
    {'name': 'github-cpython',
     'url': 'https://api.github.com/repos/python/cpython'},
    {'name': 'vendor-rates',                      # replace with your own
     'url': 'https://api.example.com/v1/rates',
     'headers': {'Authorization': 'Bearer ${VENDOR_TOKEN}'},
     'ignore': ['$.rates']},                      # a map keyed by currency code
]


def type_of(v):
    if v is None:
        return 'null'
    if isinstance(v, bool):               # check bool first: True is an int in Python
        return 'boolean'
    if isinstance(v, (int, float)):
        return 'number'
    if isinstance(v, str):
        return 'string'
    return 'array' if isinstance(v, list) else 'object'


def shape(value, ignore, path='$', out=None):
    # Reduce any JSON value to {path: {types}} - the values are thrown away.
    out = {} if out is None else out
    out.setdefault(path, set()).add(type_of(value))
    if path in ignore:                    # dynamic keys: record the node, skip its children
        return out
    if isinstance(value, dict):
        for key, child in value.items():
            shape(child, ignore, f'{path}.{key}', out)
    elif isinstance(value, list):
        for child in value:
            shape(child, ignore, f'{path}[]', out)
    return out


def fmt(types):
    return '/'.join(sorted(types))


def fetch(url, headers):
    headers = {k: os.path.expandvars(v) for k, v in headers.items()}
    req = urllib.request.Request(url, headers={'User-Agent': 'api-drift/1.0', **headers})
    with urllib.request.urlopen(req, timeout=20) as resp:
        return json.load(resp)


def check(ep, state):
    name = ep['name']
    try:
        data = fetch(ep['url'], ep.get('headers', {}))
    except Exception as err:
        return [f'{name}: request failed ({err})']

    SAMPLES.mkdir(exist_ok=True)
    (SAMPLES / f'{name}.json').write_text(json.dumps(data, indent=2))

    current = shape(data, ep.get('ignore', []))
    entry = state.get(name)
    if entry is None:                     # first run: just learn the shape
        state[name] = {'shape': {p: sorted(t) for p, t in current.items()}, 'missing': {}}
        print(f'{name}: baseline saved ({len(current)} paths)')
        return []

    known = {p: set(t) for p, t in entry['shape'].items()}
    missing = entry['missing']
    problems = []

    for path, types in current.items():
        if path not in known:
            problems.append(f'{name}: NEW {path} ({fmt(types)})')
            known[path] = set(types)
        elif types - known[path]:
            problems.append(f'{name}: TYPE {path} was {fmt(known[path])}, '
                            f'now also {fmt(types - known[path])}')
            known[path] |= types
        missing.pop(path, None)

    for path in [p for p in known if p not in current]:
        missing[path] = missing.get(path, 0) + 1
        if missing[path] >= GRACE:
            problems.append(f'{name}: REMOVED {path} (gone for {GRACE} runs)')
            del known[path]
            del missing[path]

    entry['shape'] = {p: sorted(t) for p, t in known.items()}
    return problems


def alert(problems):
    text = 'API drift detected:\n' + '\n'.join(problems)
    print(text)
    hook = os.environ.get('ALERT_WEBHOOK')   # Slack-style webhook; Discord wants 'content'
    if hook:
        req = urllib.request.Request(hook, json.dumps({'text': text}).encode(),
                                     {'Content-Type': 'application/json'})
        urllib.request.urlopen(req, timeout=10)


def save(state):
    tmp = STATE.with_suffix('.tmp')
    tmp.write_text(json.dumps(state, indent=2, sort_keys=True))
    os.replace(tmp, STATE)                # atomic: a crash can't leave half a file


def main():
    state = json.loads(STATE.read_text()) if STATE.exists() else {}
    problems = []
    for ep in ENDPOINTS:
        problems += check(ep, state)
    if problems:
        alert(problems)                   # if this raises, state isn't saved, so we alert again next run
    save(state)
    sys.exit(1 if problems else 0)


if __name__ == '__main__':
    main()

How do I set it up?

  1. Put your endpoints in ENDPOINTS. Use ${VAR} in headers so tokens stay in the environment and out of the file.

  2. Run it once by hand. The first run records a baseline for each endpoint and prints baseline saved.

  3. Schedule it. Put the secrets at the top of your crontab, then add one line:

ALERT_WEBHOOK=https://hooks.slack.com/services/XXX
VENDOR_TOKEN=your-token-here
0 7 * * * cd /home/me/api-drift && /usr/bin/python3 api_drift.py >> drift.log 2>&1

Cron uses the server's timezone, so check what 7:00 means on your machine. The script also exits with code 1 when it finds drift, so any scheduler that watches exit codes can pick it up.

What does an alert look like?

API drift detected:
vendor-rates: TYPE $.items[].price was string, now also number
vendor-rates: NEW $.items[].currency (string)
vendor-rates: REMOVED $.items[].legacy_id (gone for 3 runs)

Each line names the endpoint, the kind of change, and the exact JSON path, so you know where to look.

How does it avoid crying wolf?

Four choices keep the noise down:

  • Types, not values. Prices and timestamps can change all day without a single alert.

  • The baseline only widens. If a field is sometimes null, you get one alert the first time it happens. After that, null is part of the known shape and it stops nagging.

  • Removal needs three runs in a row. A field missing from one sample isn't proof it's gone, so GRACE waits before calling it removed.

  • Dynamic keys can be ignored. A map keyed by ID, date, or currency code would create a new path for every key. List that path in ignore and its children are skipped.

One more detail: the alert goes out before the state file is saved. If the webhook fails, the script crashes without saving, so tomorrow's run reports the same drift again instead of swallowing it.

What won't this catch?

  • Changes in meaning. If prices move from rupees to paise, they are still numbers. Units, sort order, and ID formats are invisible by design.

  • Paths your sample never touches. If you call /orders?limit=5, you won't see what /orders?status=refunded returns. Add an entry to ENDPOINTS for each variation you depend on.

  • Rarely present fields. A field that appears in one response out of a thousand may not show up in your sample for weeks. When it finally does, you get a single NEW alert.

  • Behaviour. Rate limits, pagination rules, and auth changes mostly show up as failed requests, not as shape changes.

Also, the script is silent when nothing changes, so you can't tell "no drift" from "cron stopped running". Pair it with a dead-man's-switch ping so a missing run raises its own alarm.

When I get an alert, what do I do?

  1. Open samples/<name>.json. It holds the latest raw response from that endpoint.

  2. Search your code for the path that changed and see how each place uses it.

  3. Check the provider's changelog and docs. If there is nothing, send them the sample in a support ticket.

  4. Validate responses at the edge of your code (with pydantic or jsonschema, for example) so the next surprise fails loudly in one place instead of as a blank page three layers away.

Why not use an OpenAPI diff tool instead?

If the provider publishes an OpenAPI spec, a tool like oasdiff can compare versions of it, and that is worth doing. But many APIs publish no spec, and a spec describes what the provider intends, not what the server actually returns. This script checks real responses, so it works with any JSON API and catches the cases where the docs and reality disagree.

The takeaway

You can't stop a third party from changing its API, but you can find out the next morning instead of from a customer. Diff the shape, ignore the values, alert once per change, and keep the raw sample so you can see exactly what moved.

Comments(0)

Join the discussion by logging into your account.

No comments yet. Be the first to comment!

Anshu Pathak

Passionate developer sharing knowledge about modern web technologies and best practices.

Subscribe to Anshu Pathak's Newsletter

Direct email dispatches when new stories are published. Zero algorithms.