
Sooner or later your scraper meets a 429 Too Many Requests. What it does next decides whether it is a polite client or a nuisance. A careless script retries instantly, in a tight loop, and makes the problem worse. A careful one slows down, listens to what the server says, and gives up gracefully when asked to.
This guide builds that careful behavior in four steps: don't trigger limits, read what the server tells you, wait the right amount, and know when to stop. You'll write small, tested helpers for requests, then see the same ideas in httpx and Scrapy.
One rule comes first. When a site pushes back, the right response is to slow down. Rotating proxies or user agents to get around a limit is not a fix, because it is the same traffic from more disguises (see our guides on both for the proper uses).
Version note: tested with Python 3.13,
requests2.34,urllib32.8,httpx0.28 and Scrapy 2.20.
How Servers Say "Slow Down"
Signal | What it usually means | What to do |
|---|---|---|
| You are sending requests too fast | Wait, then retry more slowly |
| Overloaded or in maintenance | Wait, then retry |
| Temporary server or gateway trouble | Retry a few times with backoff |
Timeouts, connection resets | Network or server hiccup | Retry a few times with backoff |
| Usually "you may not", not "slower" | Don't hammer it; read the response and the site's rules |
| Retrying won't change the answer | Don't retry |
The most useful signal is the Retry-After header, defined in the HTTP standard. It comes in two forms:
Retry-After: 120
Retry-After: Wed, 21 Oct 2015 07:28:00 GMTThe first is a number of seconds to wait; the second is a date after which you may try again. Your code must handle both.
Many sites also report their limits in other headers, such as X-RateLimit-Remaining. The names vary from site to site. The IETF has a draft to standardize RateLimit and RateLimit-Policy fields, but as of November 2026 it is still an Internet-Draft (revision 11, from May 2026), so check which headers your target actually sends. Retry-After is the one you can rely on.
A Server to Practice On
To test retry logic you need a server that misbehaves on demand. Save this as flaky_server.py:
import time
from email.utils import formatdate
from http.server import BaseHTTPRequestHandler, HTTPServer
hits = {}
class Flaky(BaseHTTPRequestHandler):
def do_GET(self):
n = hits[self.path] = hits.get(self.path, 0) + 1
if self.path == "/limited" and n <= 2: # too many requests, wait 2 s
self.reply(429, "slow down", {"Retry-After": "2"})
elif self.path == "/maintenance" and n == 1: # busy, back in about 3 s
when = formatdate(time.time() + 3, usegmt=True)
self.reply(503, "back soon", {"Retry-After": when})
else:
self.reply(200, f"ok (hit {n})")
def reply(self, status, text, headers=None):
body = text.encode()
self.send_response(status)
for name, value in (headers or {}).items():
self.send_header(name, value)
self.send_header("Content-Length", str(len(body)))
self.end_headers()
self.wfile.write(body)
def log_message(self, *args):
pass
HTTPServer(("127.0.0.1", 8000), Flaky).serve_forever()Run it with python flaky_server.py and install the libraries in another terminal (pip install requests httpx). Paths start working again after a couple of failures, so restart the server between experiments.
Step 1: Don't Trigger the Limit
The best retry is the one you never need. Instead of sending requests as fast as your code can loop, enforce a minimum gap per site. Better still, make the gap adaptive: widen it after pushback and narrow it gradually after success. Create backoff.py with the shared pieces first:
import email.utils
import random
import time
from datetime import datetime, timezone
from urllib.parse import urlsplit
RETRY_STATUS = {429, 500, 502, 503, 504}
MAX_WAIT = 120 # seconds: never sleep longer than this
class GaveUp(Exception):
"""Raised when a request can't be completed politely."""
def retry_after_seconds(value, now=None):
"""Turn a Retry-After header (seconds or HTTP date) into seconds, or None."""
if not value:
return None
value = value.strip()
if value.isascii() and value.isdigit():
return float(value)
try:
when = email.utils.parsedate_to_datetime(value)
except (TypeError, ValueError):
return None
if when.tzinfo is None:
when = when.replace(tzinfo=timezone.utc)
now = now or datetime.now(timezone.utc)
return max(0.0, (when - now).total_seconds())
def full_jitter(attempt, base=1.0, cap=60.0):
"""Wait a random time between 0 and an exponentially growing ceiling."""
return random.uniform(0, min(cap, base * 2 ** attempt))Then add the limiter to the same file:
class AdaptiveLimiter:
"""Keeps a minimum gap between requests to each host.
The gap doubles after pushback and shrinks slowly after successes.
"""
def __init__(self, min_gap=1.0, max_gap=30.0):
self.min_gap, self.max_gap = min_gap, max_gap
self.gap = {} # host -> current gap in seconds
self.next_at = {} # host -> earliest time.monotonic() for the next request
def wait(self, url):
host = urlsplit(url).netloc
now = time.monotonic()
start = max(now, self.next_at.get(host, 0.0))
gap = self.gap.get(host, self.min_gap)
self.next_at[host] = start + gap * random.uniform(1.0, 1.5)
if start > now:
time.sleep(start - now)
def slow_down(self, url, pause=0.0):
host = urlsplit(url).netloc
self.gap[host] = min(self.max_gap, self.gap.get(host, self.min_gap) * 2)
self.next_at[host] = max(self.next_at.get(host, 0.0), time.monotonic() + pause)
def speed_up(self, url):
host = urlsplit(url).netloc
self.gap[host] = max(self.min_gap, self.gap.get(host, self.min_gap) * 0.9)Call limiter.wait(url) before every request. The random 1.0 to 1.5 multiplier keeps your timing from looking mechanical, and speed_up recovers only 10% per success, so after trouble the scraper stays cautious for a while. The limiter isn't thread-safe, so give each thread its own or guard it with a lock.
Step 2: Read Retry-After Correctly
retry_after_seconds() above handles both header formats and returns None for junk. A few quick checks show what it does:
from datetime import datetime, timezone
from backoff import retry_after_seconds
assert retry_after_seconds("120") == 120
assert retry_after_seconds("soon") is None
assert retry_after_seconds("Wed, 21 Oct 2015 07:28:00 GMT") == 0 # a date in the past
now = datetime(2015, 10, 21, 7, 27, 0, tzinfo=timezone.utc)
assert retry_after_seconds("Wed, 21 Oct 2015 07:28:00 GMT", now=now) == 60Treat the value as a minimum. Waiting a little longer is always fine; waiting less is not.
Step 3: Back Off with Jitter
When there is no Retry-After (a 500, a timeout), you have to guess a sensible wait. Exponential backoff doubles the wait after each failure: 1 second, then 2, 4, 8, and so on up to a cap. But if many clients fail at the same moment and all use the same schedule, they all come back at the same moment too, and the server is hit by a wave again and again.
Jitter fixes that by randomizing the wait. The AWS Architecture Blog compared the common strategies, and plain backoff without jitter came out worst. The simplest good option is full jitter, which is what full_jitter() above does: wait a random time between 0 and min(cap, base * 2 ** attempt). With the defaults, the ceiling for each attempt is 1, 2, 4, 8, 16, 32 and then 60 seconds, and the actual wait is anywhere below it.
Step 4: Put It Together
Now combine the pieces in polite.py:
import logging
import random
import time
import requests
from backoff import (MAX_WAIT, RETRY_STATUS, AdaptiveLimiter, GaveUp,
full_jitter, retry_after_seconds)
log = logging.getLogger("polite")
def polite_get(session, url, limiter, max_attempts=5, **kwargs):
"""GET with rate limiting, Retry-After support and jittered backoff."""
kwargs.setdefault("timeout", 10)
reason = "no response"
for attempt in range(max_attempts):
limiter.wait(url)
try:
response = session.get(url, **kwargs)
except (requests.ConnectionError, requests.Timeout) as exc:
reason = type(exc).__name__
pause = full_jitter(attempt)
else:
if response.status_code not in RETRY_STATUS:
limiter.speed_up(url)
return response
reason = f"HTTP {response.status_code}"
pause = retry_after_seconds(response.headers.get("Retry-After"))
if pause is None:
pause = full_jitter(attempt)
elif pause > MAX_WAIT:
raise GaveUp(f"{url}: the server asked us to wait {pause:.0f}s")
else:
pause += random.uniform(0, 1) # obey it, plus a little jitter
log.info("%s failed (%s), pausing %.1fs", url, reason, pause)
limiter.slow_down(url, pause)
raise GaveUp(f"{url}: still failing after {max_attempts} attempts ({reason})")
if __name__ == "__main__":
logging.basicConfig(level=logging.INFO, format=" %(message)s")
limiter = AdaptiveLimiter(min_gap=0.5)
start = time.monotonic()
with requests.Session() as session:
for path in ("/limited", "/maintenance", "/ok"):
response = polite_get(session, f"http://127.0.0.1:8000{path}", limiter)
elapsed = time.monotonic() - start
print(f"{elapsed:5.1f}s {path:<13} -> {response.status_code} {response.text}")Run python polite.py against the flaky server:
http://127.0.0.1:8000/limited failed (HTTP 429), pausing 2.6s
http://127.0.0.1:8000/limited failed (HTTP 429), pausing 2.5s
5.1s /limited -> 200 ok (hit 3)
http://127.0.0.1:8000/maintenance failed (HTTP 503), pausing 3.0s
10.8s /maintenance -> 200 ok (hit 2)
15.6s /ok -> 200 ok (hit 1)Your timings will differ slightly because of the jitter. Notice a few things:
The pauses obey the server. The first two are
Retry-After: 2plus up to a second of jitter, and the third comes from an HTTP date.Pushback has a memory.
/okwaited almost five seconds after the previous success, because the limiter had widened its gap after three failures.It stops when asked to. A
Retry-Afterlonger thanMAX_WAITraisesGaveUpinstead of sleeping for an hour, and so does running out of attempts. The caller decides whether to skip the URL or stop the crawl.It only retries what can succeed. A
404comes straight back and is not retried.
The Built-in Option: urllib3's Retry
If you'd rather not write a loop, requests can retry through urllib3's Retry class:
import requests
from requests.adapters import HTTPAdapter
from urllib3.util import Retry
retry = Retry(
total=5,
status_forcelist=[429, 500, 502, 503, 504],
backoff_factor=1, # waits 0 s, 2 s, 4 s, 8 s ... between retries
backoff_max=60,
backoff_jitter=0.5, # adds 0 to 0.5 s on top of each wait
respect_retry_after_header=True,
retry_after_max=120, # the default is 21,600 seconds (six hours)
raise_on_status=False, # return the last response instead of raising
)
session = requests.Session()
session.mount("https://", HTTPAdapter(max_retries=retry))
session.mount("http://", HTTPAdapter(max_retries=retry))I measured how it behaves against the flaky server, and some defaults will surprise you:
Retry-Afteris honored, in both seconds and date form, for429and503once those codes are instatus_forcelist.The cap is six hours. Without
retry_after_max, a server that says "come back in a day" can freeze your scraper for hours.The first retry is immediate. With
backoff_factor=1the waits were 0, 2, 4 and 8 seconds.backoff_jitteris additive. It adds a small random amount to the wait, which is not the full jitter described above.Only idempotent methods are retried.
POSTis excluded by default.Giving up raises. When retries run out, you get a
RetryErrorunless you setraise_on_status=False.
It is concise and widely used, but it has no per-site limiter and no "stop when asked" rule, so polite_get() above gives you more control.
The Same Idea in httpx
httpx.HTTPTransport(retries=3) sounds like what you want, but it only retries failed connection attempts. I confirmed that a 500 response is not retried at all. For status codes you need your own loop, and the helpers from backoff.py slot straight in:
import time
import httpx
from backoff import (MAX_WAIT, RETRY_STATUS, GaveUp,
full_jitter, retry_after_seconds)
def get_with_retries(client, url, limiter, max_attempts=5):
for attempt in range(max_attempts):
limiter.wait(url)
try:
response = client.get(url)
except httpx.TransportError:
pause = full_jitter(attempt)
else:
if response.status_code not in RETRY_STATUS:
limiter.speed_up(url)
return response
pause = retry_after_seconds(response.headers.get("Retry-After"))
if pause is None:
pause = full_jitter(attempt)
elif pause > MAX_WAIT:
raise GaveUp(f"{url}: the server asked us to wait {pause:.0f}s")
limiter.slow_down(url, pause)
raise GaveUp(f"{url}: still failing after {max_attempts} attempts")Use it with with httpx.Client(timeout=10) as client: and the same AdaptiveLimiter.
Rate Limits and Retries in Scrapy
Scrapy has retries built in: failed requests are retried up to RETRY_TIMES (default 2) for the status codes in RETRY_HTTP_CODES (default 408, 429, 500, 502, 503, 504, 522, 524), and a retried request gets a lower priority, so it waits behind the others. AutoThrottle, which we covered in our Scrapy guide, also slows the crawl when the server slows down.
What Scrapy doesn't do is read Retry-After. In my test, a 429 that asked for four seconds was followed by the next request within about 1.5 seconds, and the crawl carried on.
A small middleware fixes that. It remembers when each site may be contacted again and holds back new requests until then. Put backoff.py inside your project package and add this to myproject/middlewares.py:
import asyncio
import logging
import time
from urllib.parse import urlsplit
from scrapy.downloadermiddlewares.retry import RetryMiddleware
from myproject.backoff import MAX_WAIT, retry_after_seconds
log = logging.getLogger(__name__)
class RetryAfterMiddleware(RetryMiddleware):
"""Retry as usual, but honor Retry-After for every request to that site."""
def __init__(self, settings):
super().__init__(settings)
self.resume_at = {} # host -> time.monotonic() when requests may resume
async def process_request(self, request):
host = urlsplit(request.url).netloc
wait = self.resume_at.get(host, 0) - time.monotonic()
if wait > 0:
await asyncio.sleep(wait)
def process_response(self, request, response):
if response.status in self.retry_http_codes:
header = response.headers.get("Retry-After", b"").decode("latin-1")
wait = retry_after_seconds(header)
if wait and wait > MAX_WAIT:
log.error("Retry-After of %.0fs is too long; stopping the crawl", wait)
self._stop_task = asyncio.create_task(self.crawler.stop_async())
return response
if wait:
host = urlsplit(request.url).netloc
resume = time.monotonic() + wait
self.resume_at[host] = max(self.resume_at.get(host, 0), resume)
return super().process_response(request, response)Then enable it in settings.py:
RETRY_TIMES = 4
CONCURRENT_REQUESTS = 1
DOWNLOADER_MIDDLEWARES = {
"scrapy.downloadermiddlewares.retry.RetryMiddleware": None, # turn off the built-in one
"myproject.middlewares.RetryAfterMiddleware": 550, # same priority, new behavior
}With this in place, the same test paused for the full four seconds, and a Retry-After of 1,000 seconds stopped the crawl cleanly instead of waiting. Two details matter:
CONCURRENT_REQUESTS = 1is deliberate. The middleware can only hold back requests that haven't entered the downloader yet. With Scrapy's default of 16, several requests are already queued and slip through. One request at a time keeps the pause honest. The new project template's limit of 1 request per domain isn't enough on its own, because the queued requests have already passed the middleware.Use
stop_async(), notstop(). In Scrapy 2.20,Crawler.stop()is deprecated.
Being a Good Citizen
Reduce the load you create. Don't fetch pages you already have.
Start gentle. One request per second or two is a reasonable opening pace for a small site, and five attempts with a two-minute cap is a reasonable retry budget. These are starting points, not rules, so adjust them to what the site tells you.
Check for an API. Official APIs publish their limits and are meant to be called.
Our guide "Is Web Scraping Legal?" covers the wider picture of terms of service and good practice.
Troubleshooting
Symptom | Likely cause | What to try |
|---|---|---|
Still getting | Your base rate is too high | Raise |
Scraper froze for hours | A huge | Set |
Everything retries at the same instant | No jitter | Use |
Scrapy ignores | The built-in retry middleware doesn't read it | Use |
Retries never succeed | A permanent failure ( | Remove it; don't retry those |
| Some sites block with a normal status | Check the page content, not just the status code |
Crawl stays slow after one | The limiter recovers by 10% per success | Expected; it speeds up again over time |
Where to Go from Here
You now have a client that slows itself down, obeys the server, backs off with jitter and knows when to quit. To build on it:
Run it on a schedule so it works at quiet times.
Cache responses with
ETagorLast-Modifiedso unchanged pages cost nothing.
Happy scraping, and please do it politely!
Found this helpful? Share it with a fellow developer or leave your questions in the comments.
Comments(0)
Join the discussion by logging into your account.
No comments yet. Be the first to comment!