--- title: "Handling Rate Limits and Retries in Python: A Guide to Polite Scraping" author: "Bhavya Arora" published: "2026-11-19T07:24:36.762Z" updated: "2026-11-19T07:24:36.762Z" url: "https://zyvop.com/handling-rate-limits-and-retries-in-python-a-guide-to-polite-scraping-1udxe" canonical: "https://zyvop.com/handling-rate-limits-and-retries-in-python-a-guide-to-polite-scraping-1udxe" tags: ["Web Scraping","Python","Retries","Rate Limiting","http"] --- # Handling Rate Limits and Retries in Python: A Guide to Polite Scraping Stop hammering servers. Build a scraper that slows itself down, obeys Retry-After, backs off with jitter and knows when to give up. ![Cover image]() Sooner or later your scraper meets a `429 Too Many Requests`. What it does next decides whether it is a polite client or a nuisance. A careless script retries instantly, in a tight loop, and makes the problem worse. A careful one slows down, listens to what the server says, and gives up gracefully when asked to. This guide builds that careful behavior in four steps: **don't trigger limits, read what the server tells you, wait the right amount, and know when to stop.** You'll write small, tested helpers for `requests`, then see the same ideas in `httpx` and Scrapy. One rule comes first. When a site pushes back, the right response is to **slow down**. Rotating proxies or user agents to get around a limit is not a fix, because it is the same traffic from more disguises (see our guides on both for the proper uses). > **Version note:** tested with Python 3.13, `requests` 2.34, `urllib3` 2.8, `httpx` 0.28 and Scrapy 2.20. * * * ## How Servers Say "Slow Down" | Signal | What it usually means | What to do | | --- | --- | --- | | `429 Too Many Requests` | You are sending requests too fast | Wait, then retry more slowly | | `503 Service Unavailable` | Overloaded or in maintenance | Wait, then retry | | `500`, `502`, `504` | Temporary server or gateway trouble | Retry a few times with backoff | | Timeouts, connection resets | Network or server hiccup | Retry a few times with backoff | | `403 Forbidden` | Usually "you may not", not "slower" | Don't hammer it; read the response and the site's rules | | `400`, `401`, `404` | Retrying won't change the answer | Don't retry | The most useful signal is the `Retry-After` header, defined in the HTTP standard. It comes in two forms: ```http Retry-After: 120 Retry-After: Wed, 21 Oct 2015 07:28:00 GMT ``` The first is a number of seconds to wait; the second is a date after which you may try again. Your code must handle both. Many sites also report their limits in other headers, such as `X-RateLimit-Remaining`. The names vary from site to site. The IETF has a draft to standardize `RateLimit` and `RateLimit-Policy` fields, but as of November 2026 it is still an Internet-Draft (revision 11, from May 2026), so check which headers your target actually sends. `Retry-After` is the one you can rely on. ## A Server to Practice On To test retry logic you need a server that misbehaves on demand. Save this as `flaky_server.py`: ```python import time from email.utils import formatdate from http.server import BaseHTTPRequestHandler, HTTPServer hits = {} class Flaky(BaseHTTPRequestHandler): def do_GET(self): n = hits[self.path] = hits.get(self.path, 0) + 1 if self.path == "/limited" and n <= 2: # too many requests, wait 2 s self.reply(429, "slow down", {"Retry-After": "2"}) elif self.path == "/maintenance" and n == 1: # busy, back in about 3 s when = formatdate(time.time() + 3, usegmt=True) self.reply(503, "back soon", {"Retry-After": when}) else: self.reply(200, f"ok (hit {n})") def reply(self, status, text, headers=None): body = text.encode() self.send_response(status) for name, value in (headers or {}).items(): self.send_header(name, value) self.send_header("Content-Length", str(len(body))) self.end_headers() self.wfile.write(body) def log_message(self, *args): pass HTTPServer(("127.0.0.1", 8000), Flaky).serve_forever() ``` Run it with `python flaky_server.py` and install the libraries in another terminal (`pip install requests httpx`). Paths start working again after a couple of failures, so restart the server between experiments. ## Step 1: Don't Trigger the Limit The best retry is the one you never need. Instead of sending requests as fast as your code can loop, enforce a minimum gap per site. Better still, make the gap **adaptive**: widen it after pushback and narrow it gradually after success. Create `backoff.py` with the shared pieces first: ```python import email.utils import random import time from datetime import datetime, timezone from urllib.parse import urlsplit RETRY_STATUS = {429, 500, 502, 503, 504} MAX_WAIT = 120 # seconds: never sleep longer than this class GaveUp(Exception): """Raised when a request can't be completed politely.""" def retry_after_seconds(value, now=None): """Turn a Retry-After header (seconds or HTTP date) into seconds, or None.""" if not value: return None value = value.strip() if value.isascii() and value.isdigit(): return float(value) try: when = email.utils.parsedate_to_datetime(value) except (TypeError, ValueError): return None if when.tzinfo is None: when = when.replace(tzinfo=timezone.utc) now = now or datetime.now(timezone.utc) return max(0.0, (when - now).total_seconds()) def full_jitter(attempt, base=1.0, cap=60.0): """Wait a random time between 0 and an exponentially growing ceiling.""" return random.uniform(0, min(cap, base * 2 ** attempt)) ``` Then add the limiter to the same file: ```python class AdaptiveLimiter: """Keeps a minimum gap between requests to each host. The gap doubles after pushback and shrinks slowly after successes. """ def __init__(self, min_gap=1.0, max_gap=30.0): self.min_gap, self.max_gap = min_gap, max_gap self.gap = {} # host -> current gap in seconds self.next_at = {} # host -> earliest time.monotonic() for the next request def wait(self, url): host = urlsplit(url).netloc now = time.monotonic() start = max(now, self.next_at.get(host, 0.0)) gap = self.gap.get(host, self.min_gap) self.next_at[host] = start + gap * random.uniform(1.0, 1.5) if start > now: time.sleep(start - now) def slow_down(self, url, pause=0.0): host = urlsplit(url).netloc self.gap[host] = min(self.max_gap, self.gap.get(host, self.min_gap) * 2) self.next_at[host] = max(self.next_at.get(host, 0.0), time.monotonic() + pause) def speed_up(self, url): host = urlsplit(url).netloc self.gap[host] = max(self.min_gap, self.gap.get(host, self.min_gap) * 0.9) ``` Call `limiter.wait(url)` before every request. The random 1.0 to 1.5 multiplier keeps your timing from looking mechanical, and `speed_up` recovers only 10% per success, so after trouble the scraper stays cautious for a while. The limiter isn't thread-safe, so give each thread its own or guard it with a lock. ## Step 2: Read Retry-After Correctly `retry_after_seconds()` above handles both header formats and returns `None` for junk. A few quick checks show what it does: ```python from datetime import datetime, timezone from backoff import retry_after_seconds assert retry_after_seconds("120") == 120 assert retry_after_seconds("soon") is None assert retry_after_seconds("Wed, 21 Oct 2015 07:28:00 GMT") == 0 # a date in the past now = datetime(2015, 10, 21, 7, 27, 0, tzinfo=timezone.utc) assert retry_after_seconds("Wed, 21 Oct 2015 07:28:00 GMT", now=now) == 60 ``` Treat the value as a **minimum**. Waiting a little longer is always fine; waiting less is not. ## Step 3: Back Off with Jitter When there is no `Retry-After` (a `500`, a timeout), you have to guess a sensible wait. **Exponential backoff** doubles the wait after each failure: 1 second, then 2, 4, 8, and so on up to a cap. But if many clients fail at the same moment and all use the same schedule, they all come back at the same moment too, and the server is hit by a wave again and again. **Jitter** fixes that by randomizing the wait. The AWS Architecture Blog compared the common strategies, and plain backoff without jitter came out worst. The simplest good option is **full jitter**, which is what `full_jitter()` above does: wait a random time between 0 and `min(cap, base * 2 ** attempt)`. With the defaults, the ceiling for each attempt is 1, 2, 4, 8, 16, 32 and then 60 seconds, and the actual wait is anywhere below it. ## Step 4: Put It Together Now combine the pieces in `polite.py`: ```python import logging import random import time import requests from backoff import (MAX_WAIT, RETRY_STATUS, AdaptiveLimiter, GaveUp, full_jitter, retry_after_seconds) log = logging.getLogger("polite") def polite_get(session, url, limiter, max_attempts=5, **kwargs): """GET with rate limiting, Retry-After support and jittered backoff.""" kwargs.setdefault("timeout", 10) reason = "no response" for attempt in range(max_attempts): limiter.wait(url) try: response = session.get(url, **kwargs) except (requests.ConnectionError, requests.Timeout) as exc: reason = type(exc).__name__ pause = full_jitter(attempt) else: if response.status_code not in RETRY_STATUS: limiter.speed_up(url) return response reason = f"HTTP {response.status_code}" pause = retry_after_seconds(response.headers.get("Retry-After")) if pause is None: pause = full_jitter(attempt) elif pause > MAX_WAIT: raise GaveUp(f"{url}: the server asked us to wait {pause:.0f}s") else: pause += random.uniform(0, 1) # obey it, plus a little jitter log.info("%s failed (%s), pausing %.1fs", url, reason, pause) limiter.slow_down(url, pause) raise GaveUp(f"{url}: still failing after {max_attempts} attempts ({reason})") if __name__ == "__main__": logging.basicConfig(level=logging.INFO, format=" %(message)s") limiter = AdaptiveLimiter(min_gap=0.5) start = time.monotonic() with requests.Session() as session: for path in ("/limited", "/maintenance", "/ok"): response = polite_get(session, f"http://127.0.0.1:8000{path}", limiter) elapsed = time.monotonic() - start print(f"{elapsed:5.1f}s {path:<13} -> {response.status_code} {response.text}") ``` Run `python polite.py` against the flaky server: ```text http://127.0.0.1:8000/limited failed (HTTP 429), pausing 2.6s http://127.0.0.1:8000/limited failed (HTTP 429), pausing 2.5s 5.1s /limited -> 200 ok (hit 3) http://127.0.0.1:8000/maintenance failed (HTTP 503), pausing 3.0s 10.8s /maintenance -> 200 ok (hit 2) 15.6s /ok -> 200 ok (hit 1) ``` Your timings will differ slightly because of the jitter. Notice a few things: - **The pauses obey the server.** The first two are `Retry-After: 2` plus up to a second of jitter, and the third comes from an HTTP date. - **Pushback has a memory.** `/ok` waited almost five seconds after the previous success, because the limiter had widened its gap after three failures. - **It stops when asked to.** A `Retry-After` longer than `MAX_WAIT` raises `GaveUp` instead of sleeping for an hour, and so does running out of attempts. The caller decides whether to skip the URL or stop the crawl. - **It only retries what can succeed.** A `404` comes straight back and is not retried. ## The Built-in Option: urllib3's Retry If you'd rather not write a loop, `requests` can retry through urllib3's `Retry` class: ```python import requests from requests.adapters import HTTPAdapter from urllib3.util import Retry retry = Retry( total=5, status_forcelist=[429, 500, 502, 503, 504], backoff_factor=1, # waits 0 s, 2 s, 4 s, 8 s ... between retries backoff_max=60, backoff_jitter=0.5, # adds 0 to 0.5 s on top of each wait respect_retry_after_header=True, retry_after_max=120, # the default is 21,600 seconds (six hours) raise_on_status=False, # return the last response instead of raising ) session = requests.Session() session.mount("https://", HTTPAdapter(max_retries=retry)) session.mount("http://", HTTPAdapter(max_retries=retry)) ``` I measured how it behaves against the flaky server, and some defaults will surprise you: - `Retry-After` **is honored**, in both seconds and date form, for `429` and `503` once those codes are in `status_forcelist`. - **The cap is six hours.** Without `retry_after_max`, a server that says "come back in a day" can freeze your scraper for hours. - **The first retry is immediate.** With `backoff_factor=1` the waits were 0, 2, 4 and 8 seconds. - `backoff_jitter` **is additive.** It adds a small random amount to the wait, which is not the full jitter described above. - **Only idempotent methods are retried.** `POST` is excluded by default. - **Giving up raises.** When retries run out, you get a `RetryError` unless you set `raise_on_status=False`. It is concise and widely used, but it has no per-site limiter and no "stop when asked" rule, so `polite_get()` above gives you more control. ## The Same Idea in httpx `httpx.HTTPTransport(retries=3)` sounds like what you want, but it only retries failed connection attempts. I confirmed that a `500` response is not retried at all. For status codes you need your own loop, and the helpers from `backoff.py` slot straight in: ```python import time import httpx from backoff import (MAX_WAIT, RETRY_STATUS, GaveUp, full_jitter, retry_after_seconds) def get_with_retries(client, url, limiter, max_attempts=5): for attempt in range(max_attempts): limiter.wait(url) try: response = client.get(url) except httpx.TransportError: pause = full_jitter(attempt) else: if response.status_code not in RETRY_STATUS: limiter.speed_up(url) return response pause = retry_after_seconds(response.headers.get("Retry-After")) if pause is None: pause = full_jitter(attempt) elif pause > MAX_WAIT: raise GaveUp(f"{url}: the server asked us to wait {pause:.0f}s") limiter.slow_down(url, pause) raise GaveUp(f"{url}: still failing after {max_attempts} attempts") ``` Use it with `with httpx.Client(timeout=10) as client:` and the same `AdaptiveLimiter`. ## Rate Limits and Retries in Scrapy Scrapy has retries built in: failed requests are retried up to `RETRY_TIMES` (default 2) for the status codes in `RETRY_HTTP_CODES` (default `408, 429, 500, 502, 503, 504, 522, 524`), and a retried request gets a lower priority, so it waits behind the others. AutoThrottle, which we covered in our Scrapy guide, also slows the crawl when the server slows down. What Scrapy **doesn't** do is read `Retry-After`. In my test, a `429` that asked for four seconds was followed by the next request within about 1.5 seconds, and the crawl carried on. A small middleware fixes that. It remembers when each site may be contacted again and holds back new requests until then. Put `backoff.py` inside your project package and add this to `myproject/middlewares.py`: ```python import asyncio import logging import time from urllib.parse import urlsplit from scrapy.downloadermiddlewares.retry import RetryMiddleware from myproject.backoff import MAX_WAIT, retry_after_seconds log = logging.getLogger(__name__) class RetryAfterMiddleware(RetryMiddleware): """Retry as usual, but honor Retry-After for every request to that site.""" def __init__(self, settings): super().__init__(settings) self.resume_at = {} # host -> time.monotonic() when requests may resume async def process_request(self, request): host = urlsplit(request.url).netloc wait = self.resume_at.get(host, 0) - time.monotonic() if wait > 0: await asyncio.sleep(wait) def process_response(self, request, response): if response.status in self.retry_http_codes: header = response.headers.get("Retry-After", b"").decode("latin-1") wait = retry_after_seconds(header) if wait and wait > MAX_WAIT: log.error("Retry-After of %.0fs is too long; stopping the crawl", wait) self._stop_task = asyncio.create_task(self.crawler.stop_async()) return response if wait: host = urlsplit(request.url).netloc resume = time.monotonic() + wait self.resume_at[host] = max(self.resume_at.get(host, 0), resume) return super().process_response(request, response) ``` Then enable it in `settings.py`: ```python RETRY_TIMES = 4 CONCURRENT_REQUESTS = 1 DOWNLOADER_MIDDLEWARES = { "scrapy.downloadermiddlewares.retry.RetryMiddleware": None, # turn off the built-in one "myproject.middlewares.RetryAfterMiddleware": 550, # same priority, new behavior } ``` With this in place, the same test paused for the full four seconds, and a `Retry-After` of 1,000 seconds stopped the crawl cleanly instead of waiting. Two details matter: - `CONCURRENT_REQUESTS = 1` **is deliberate.** The middleware can only hold back requests that haven't entered the downloader yet. With Scrapy's default of 16, several requests are already queued and slip through. One request at a time keeps the pause honest. The new project template's limit of 1 request per domain isn't enough on its own, because the queued requests have already passed the middleware. - **Use** `stop_async()`**, not** `stop()`**.** In Scrapy 2.20, `Crawler.stop()` is deprecated. ## Being a Good Citizen - **Reduce the load you create.** Don't fetch pages you already have. - **Start gentle.** One request per second or two is a reasonable opening pace for a small site, and five attempts with a two-minute cap is a reasonable retry budget. These are starting points, not rules, so adjust them to what the site tells you. - **Check for an API.** Official APIs publish their limits and are meant to be called. Our guide "Is Web Scraping Legal?" covers the wider picture of terms of service and good practice. ## Troubleshooting | Symptom | Likely cause | What to try | | --- | --- | --- | | Still getting `429` despite retries | Your base rate is too high | Raise `min_gap` or lower concurrency | | Scraper froze for hours | A huge `Retry-After` with no cap | Set `MAX_WAIT`, or `retry_after_max` in urllib3 | | Everything retries at the same instant | No jitter | Use `full_jitter()` | | Scrapy ignores `Retry-After` | The built-in retry middleware doesn't read it | Use `RetryAfterMiddleware` with `CONCURRENT_REQUESTS = 1` | | Retries never succeed | A permanent failure (`403`, `404`) is in your retry list | Remove it; don't retry those | | `200` responses full of "access denied" | Some sites block with a normal status | Check the page content, not just the status code | | Crawl stays slow after one `429` | The limiter recovers by 10% per success | Expected; it speeds up again over time | ## Where to Go from Here You now have a client that slows itself down, obeys the server, backs off with jitter and knows when to quit. To build on it: - Run it on a schedule so it works at quiet times. - Cache responses with `ETag` or `Last-Modified` so unchanged pages cost nothing. Happy scraping, and please do it politely! * * * _Found this helpful? Share it with a fellow developer or leave your questions in the comments._