Web Development

Handling Rate Limits and Retries in Python: A Guide to Polite Scraping

Stop hammering servers. Build a scraper that slows itself down, obeys Retry-After, backs off with jitter and knows when to give up.

·
10 min read
·
AI-assisted

AI tooling disclosure

Generative AI materially contributed to this article.

Human review confirmed by the author.

Handling Rate Limits and Retries in Python: A Guide to Polite Scraping

Sooner or later your scraper meets a 429 Too Many Requests. What it does next decides whether it is a polite client or a nuisance. A careless script retries instantly, in a tight loop, and makes the problem worse. A careful one slows down, listens to what the server says, and gives up gracefully when asked to.

This guide builds that careful behavior in four steps: don't trigger limits, read what the server tells you, wait the right amount, and know when to stop. You'll write small, tested helpers for requests, then see the same ideas in httpx and Scrapy.

One rule comes first. When a site pushes back, the right response is to slow down. Rotating proxies or user agents to get around a limit is not a fix, because it is the same traffic from more disguises (see our guides on both for the proper uses).

Version note: tested with Python 3.13, requests 2.34, urllib3 2.8, httpx 0.28 and Scrapy 2.20.


How Servers Say "Slow Down"

Signal

What it usually means

What to do

429 Too Many Requests

You are sending requests too fast

Wait, then retry more slowly

503 Service Unavailable

Overloaded or in maintenance

Wait, then retry

500, 502, 504

Temporary server or gateway trouble

Retry a few times with backoff

Timeouts, connection resets

Network or server hiccup

Retry a few times with backoff

403 Forbidden

Usually "you may not", not "slower"

Don't hammer it; read the response and the site's rules

400, 401, 404

Retrying won't change the answer

Don't retry

The most useful signal is the Retry-After header, defined in the HTTP standard. It comes in two forms:

Retry-After: 120
Retry-After: Wed, 21 Oct 2015 07:28:00 GMT

The first is a number of seconds to wait; the second is a date after which you may try again. Your code must handle both.

Many sites also report their limits in other headers, such as X-RateLimit-Remaining. The names vary from site to site. The IETF has a draft to standardize RateLimit and RateLimit-Policy fields, but as of November 2026 it is still an Internet-Draft (revision 11, from May 2026), so check which headers your target actually sends. Retry-After is the one you can rely on.

A Server to Practice On

To test retry logic you need a server that misbehaves on demand. Save this as flaky_server.py:

import time
from email.utils import formatdate
from http.server import BaseHTTPRequestHandler, HTTPServer

hits = {}


class Flaky(BaseHTTPRequestHandler):
    def do_GET(self):
        n = hits[self.path] = hits.get(self.path, 0) + 1
        if self.path == "/limited" and n <= 2:          # too many requests, wait 2 s
            self.reply(429, "slow down", {"Retry-After": "2"})
        elif self.path == "/maintenance" and n == 1:   # busy, back in about 3 s
            when = formatdate(time.time() + 3, usegmt=True)
            self.reply(503, "back soon", {"Retry-After": when})
        else:
            self.reply(200, f"ok (hit {n})")

    def reply(self, status, text, headers=None):
        body = text.encode()
        self.send_response(status)
        for name, value in (headers or {}).items():
            self.send_header(name, value)
        self.send_header("Content-Length", str(len(body)))
        self.end_headers()
        self.wfile.write(body)

    def log_message(self, *args):
        pass


HTTPServer(("127.0.0.1", 8000), Flaky).serve_forever()

Run it with python flaky_server.py and install the libraries in another terminal (pip install requests httpx). Paths start working again after a couple of failures, so restart the server between experiments.

Step 1: Don't Trigger the Limit

The best retry is the one you never need. Instead of sending requests as fast as your code can loop, enforce a minimum gap per site. Better still, make the gap adaptive: widen it after pushback and narrow it gradually after success. Create backoff.py with the shared pieces first:

import email.utils
import random
import time
from datetime import datetime, timezone
from urllib.parse import urlsplit

RETRY_STATUS = {429, 500, 502, 503, 504}
MAX_WAIT = 120          # seconds: never sleep longer than this


class GaveUp(Exception):
    """Raised when a request can't be completed politely."""


def retry_after_seconds(value, now=None):
    """Turn a Retry-After header (seconds or HTTP date) into seconds, or None."""
    if not value:
        return None
    value = value.strip()
    if value.isascii() and value.isdigit():
        return float(value)
    try:
        when = email.utils.parsedate_to_datetime(value)
    except (TypeError, ValueError):
        return None
    if when.tzinfo is None:
        when = when.replace(tzinfo=timezone.utc)
    now = now or datetime.now(timezone.utc)
    return max(0.0, (when - now).total_seconds())


def full_jitter(attempt, base=1.0, cap=60.0):
    """Wait a random time between 0 and an exponentially growing ceiling."""
    return random.uniform(0, min(cap, base * 2 ** attempt))

Then add the limiter to the same file:

class AdaptiveLimiter:
    """Keeps a minimum gap between requests to each host.

    The gap doubles after pushback and shrinks slowly after successes.
    """

    def __init__(self, min_gap=1.0, max_gap=30.0):
        self.min_gap, self.max_gap = min_gap, max_gap
        self.gap = {}        # host -> current gap in seconds
        self.next_at = {}    # host -> earliest time.monotonic() for the next request

    def wait(self, url):
        host = urlsplit(url).netloc
        now = time.monotonic()
        start = max(now, self.next_at.get(host, 0.0))
        gap = self.gap.get(host, self.min_gap)
        self.next_at[host] = start + gap * random.uniform(1.0, 1.5)
        if start > now:
            time.sleep(start - now)

    def slow_down(self, url, pause=0.0):
        host = urlsplit(url).netloc
        self.gap[host] = min(self.max_gap, self.gap.get(host, self.min_gap) * 2)
        self.next_at[host] = max(self.next_at.get(host, 0.0), time.monotonic() + pause)

    def speed_up(self, url):
        host = urlsplit(url).netloc
        self.gap[host] = max(self.min_gap, self.gap.get(host, self.min_gap) * 0.9)

Call limiter.wait(url) before every request. The random 1.0 to 1.5 multiplier keeps your timing from looking mechanical, and speed_up recovers only 10% per success, so after trouble the scraper stays cautious for a while. The limiter isn't thread-safe, so give each thread its own or guard it with a lock.

Step 2: Read Retry-After Correctly

retry_after_seconds() above handles both header formats and returns None for junk. A few quick checks show what it does:

from datetime import datetime, timezone

from backoff import retry_after_seconds

assert retry_after_seconds("120") == 120
assert retry_after_seconds("soon") is None
assert retry_after_seconds("Wed, 21 Oct 2015 07:28:00 GMT") == 0     # a date in the past

now = datetime(2015, 10, 21, 7, 27, 0, tzinfo=timezone.utc)
assert retry_after_seconds("Wed, 21 Oct 2015 07:28:00 GMT", now=now) == 60

Treat the value as a minimum. Waiting a little longer is always fine; waiting less is not.

Step 3: Back Off with Jitter

When there is no Retry-After (a 500, a timeout), you have to guess a sensible wait. Exponential backoff doubles the wait after each failure: 1 second, then 2, 4, 8, and so on up to a cap. But if many clients fail at the same moment and all use the same schedule, they all come back at the same moment too, and the server is hit by a wave again and again.

Jitter fixes that by randomizing the wait. The AWS Architecture Blog compared the common strategies, and plain backoff without jitter came out worst. The simplest good option is full jitter, which is what full_jitter() above does: wait a random time between 0 and min(cap, base * 2 ** attempt). With the defaults, the ceiling for each attempt is 1, 2, 4, 8, 16, 32 and then 60 seconds, and the actual wait is anywhere below it.

Step 4: Put It Together

Now combine the pieces in polite.py:

import logging
import random
import time

import requests

from backoff import (MAX_WAIT, RETRY_STATUS, AdaptiveLimiter, GaveUp,
                     full_jitter, retry_after_seconds)

log = logging.getLogger("polite")


def polite_get(session, url, limiter, max_attempts=5, **kwargs):
    """GET with rate limiting, Retry-After support and jittered backoff."""
    kwargs.setdefault("timeout", 10)
    reason = "no response"
    for attempt in range(max_attempts):
        limiter.wait(url)
        try:
            response = session.get(url, **kwargs)
        except (requests.ConnectionError, requests.Timeout) as exc:
            reason = type(exc).__name__
            pause = full_jitter(attempt)
        else:
            if response.status_code not in RETRY_STATUS:
                limiter.speed_up(url)
                return response
            reason = f"HTTP {response.status_code}"
            pause = retry_after_seconds(response.headers.get("Retry-After"))
            if pause is None:
                pause = full_jitter(attempt)
            elif pause > MAX_WAIT:
                raise GaveUp(f"{url}: the server asked us to wait {pause:.0f}s")
            else:
                pause += random.uniform(0, 1)       # obey it, plus a little jitter
        log.info("%s failed (%s), pausing %.1fs", url, reason, pause)
        limiter.slow_down(url, pause)
    raise GaveUp(f"{url}: still failing after {max_attempts} attempts ({reason})")


if __name__ == "__main__":
    logging.basicConfig(level=logging.INFO, format="  %(message)s")
    limiter = AdaptiveLimiter(min_gap=0.5)
    start = time.monotonic()
    with requests.Session() as session:
        for path in ("/limited", "/maintenance", "/ok"):
            response = polite_get(session, f"http://127.0.0.1:8000{path}", limiter)
            elapsed = time.monotonic() - start
            print(f"{elapsed:5.1f}s  {path:<13} -> {response.status_code} {response.text}")

Run python polite.py against the flaky server:

http://127.0.0.1:8000/limited failed (HTTP 429), pausing 2.6s
  http://127.0.0.1:8000/limited failed (HTTP 429), pausing 2.5s
  5.1s  /limited      -> 200 ok (hit 3)
  http://127.0.0.1:8000/maintenance failed (HTTP 503), pausing 3.0s
 10.8s  /maintenance  -> 200 ok (hit 2)
 15.6s  /ok           -> 200 ok (hit 1)

Your timings will differ slightly because of the jitter. Notice a few things:

  • The pauses obey the server. The first two are Retry-After: 2 plus up to a second of jitter, and the third comes from an HTTP date.

  • Pushback has a memory. /ok waited almost five seconds after the previous success, because the limiter had widened its gap after three failures.

  • It stops when asked to. A Retry-After longer than MAX_WAIT raises GaveUp instead of sleeping for an hour, and so does running out of attempts. The caller decides whether to skip the URL or stop the crawl.

  • It only retries what can succeed. A 404 comes straight back and is not retried.

The Built-in Option: urllib3's Retry

If you'd rather not write a loop, requests can retry through urllib3's Retry class:

import requests
from requests.adapters import HTTPAdapter
from urllib3.util import Retry

retry = Retry(
    total=5,
    status_forcelist=[429, 500, 502, 503, 504],
    backoff_factor=1,        # waits 0 s, 2 s, 4 s, 8 s ... between retries
    backoff_max=60,
    backoff_jitter=0.5,      # adds 0 to 0.5 s on top of each wait
    respect_retry_after_header=True,
    retry_after_max=120,     # the default is 21,600 seconds (six hours)
    raise_on_status=False,   # return the last response instead of raising
)

session = requests.Session()
session.mount("https://", HTTPAdapter(max_retries=retry))
session.mount("http://", HTTPAdapter(max_retries=retry))

I measured how it behaves against the flaky server, and some defaults will surprise you:

  • Retry-After is honored, in both seconds and date form, for 429 and 503 once those codes are in status_forcelist.

  • The cap is six hours. Without retry_after_max, a server that says "come back in a day" can freeze your scraper for hours.

  • The first retry is immediate. With backoff_factor=1 the waits were 0, 2, 4 and 8 seconds.

  • backoff_jitter is additive. It adds a small random amount to the wait, which is not the full jitter described above.

  • Only idempotent methods are retried. POST is excluded by default.

  • Giving up raises. When retries run out, you get a RetryError unless you set raise_on_status=False.

It is concise and widely used, but it has no per-site limiter and no "stop when asked" rule, so polite_get() above gives you more control.

The Same Idea in httpx

httpx.HTTPTransport(retries=3) sounds like what you want, but it only retries failed connection attempts. I confirmed that a 500 response is not retried at all. For status codes you need your own loop, and the helpers from backoff.py slot straight in:

import time

import httpx

from backoff import (MAX_WAIT, RETRY_STATUS, GaveUp,
                     full_jitter, retry_after_seconds)


def get_with_retries(client, url, limiter, max_attempts=5):
    for attempt in range(max_attempts):
        limiter.wait(url)
        try:
            response = client.get(url)
        except httpx.TransportError:
            pause = full_jitter(attempt)
        else:
            if response.status_code not in RETRY_STATUS:
                limiter.speed_up(url)
                return response
            pause = retry_after_seconds(response.headers.get("Retry-After"))
            if pause is None:
                pause = full_jitter(attempt)
            elif pause > MAX_WAIT:
                raise GaveUp(f"{url}: the server asked us to wait {pause:.0f}s")
        limiter.slow_down(url, pause)
    raise GaveUp(f"{url}: still failing after {max_attempts} attempts")

Use it with with httpx.Client(timeout=10) as client: and the same AdaptiveLimiter.

Rate Limits and Retries in Scrapy

Scrapy has retries built in: failed requests are retried up to RETRY_TIMES (default 2) for the status codes in RETRY_HTTP_CODES (default 408, 429, 500, 502, 503, 504, 522, 524), and a retried request gets a lower priority, so it waits behind the others. AutoThrottle, which we covered in our Scrapy guide, also slows the crawl when the server slows down.

What Scrapy doesn't do is read Retry-After. In my test, a 429 that asked for four seconds was followed by the next request within about 1.5 seconds, and the crawl carried on.

A small middleware fixes that. It remembers when each site may be contacted again and holds back new requests until then. Put backoff.py inside your project package and add this to myproject/middlewares.py:

import asyncio
import logging
import time
from urllib.parse import urlsplit

from scrapy.downloadermiddlewares.retry import RetryMiddleware

from myproject.backoff import MAX_WAIT, retry_after_seconds

log = logging.getLogger(__name__)


class RetryAfterMiddleware(RetryMiddleware):
    """Retry as usual, but honor Retry-After for every request to that site."""

    def __init__(self, settings):
        super().__init__(settings)
        self.resume_at = {}      # host -> time.monotonic() when requests may resume

    async def process_request(self, request):
        host = urlsplit(request.url).netloc
        wait = self.resume_at.get(host, 0) - time.monotonic()
        if wait > 0:
            await asyncio.sleep(wait)

    def process_response(self, request, response):
        if response.status in self.retry_http_codes:
            header = response.headers.get("Retry-After", b"").decode("latin-1")
            wait = retry_after_seconds(header)
            if wait and wait > MAX_WAIT:
                log.error("Retry-After of %.0fs is too long; stopping the crawl", wait)
                self._stop_task = asyncio.create_task(self.crawler.stop_async())
                return response
            if wait:
                host = urlsplit(request.url).netloc
                resume = time.monotonic() + wait
                self.resume_at[host] = max(self.resume_at.get(host, 0), resume)
        return super().process_response(request, response)

Then enable it in settings.py:

RETRY_TIMES = 4
CONCURRENT_REQUESTS = 1

DOWNLOADER_MIDDLEWARES = {
    "scrapy.downloadermiddlewares.retry.RetryMiddleware": None,    # turn off the built-in one
    "myproject.middlewares.RetryAfterMiddleware": 550,             # same priority, new behavior
}

With this in place, the same test paused for the full four seconds, and a Retry-After of 1,000 seconds stopped the crawl cleanly instead of waiting. Two details matter:

  • CONCURRENT_REQUESTS = 1 is deliberate. The middleware can only hold back requests that haven't entered the downloader yet. With Scrapy's default of 16, several requests are already queued and slip through. One request at a time keeps the pause honest. The new project template's limit of 1 request per domain isn't enough on its own, because the queued requests have already passed the middleware.

  • Use stop_async(), not stop(). In Scrapy 2.20, Crawler.stop() is deprecated.

Being a Good Citizen

  • Reduce the load you create. Don't fetch pages you already have.

  • Start gentle. One request per second or two is a reasonable opening pace for a small site, and five attempts with a two-minute cap is a reasonable retry budget. These are starting points, not rules, so adjust them to what the site tells you.

  • Check for an API. Official APIs publish their limits and are meant to be called.

Our guide "Is Web Scraping Legal?" covers the wider picture of terms of service and good practice.

Troubleshooting

Symptom

Likely cause

What to try

Still getting 429 despite retries

Your base rate is too high

Raise min_gap or lower concurrency

Scraper froze for hours

A huge Retry-After with no cap

Set MAX_WAIT, or retry_after_max in urllib3

Everything retries at the same instant

No jitter

Use full_jitter()

Scrapy ignores Retry-After

The built-in retry middleware doesn't read it

Use RetryAfterMiddleware with CONCURRENT_REQUESTS = 1

Retries never succeed

A permanent failure (403, 404) is in your retry list

Remove it; don't retry those

200 responses full of "access denied"

Some sites block with a normal status

Check the page content, not just the status code

Crawl stays slow after one 429

The limiter recovers by 10% per success

Expected; it speeds up again over time

Where to Go from Here

You now have a client that slows itself down, obeys the server, backs off with jitter and knows when to quit. To build on it:

  • Run it on a schedule so it works at quiet times.

  • Cache responses with ETag or Last-Modified so unchanged pages cost nothing.

Happy scraping, and please do it politely!


Found this helpful? Share it with a fellow developer or leave your questions in the comments.

Comments(0)

Join the discussion by logging into your account.

No comments yet. Be the first to comment!

Bhavya Arora

Passionate developer sharing knowledge about modern web technologies and best practices.

Subscribe to Bhavya Arora's Newsletter

Direct email dispatches when new stories are published. Zero algorithms.