{"schemaVersion":"1.0","type":"TechArticle","types":["Article","TechArticle"],"slug":"handling-rate-limits-and-retries-in-python-a-guide-to-polite-scraping-1udxe","url":"https://zyvop.com/handling-rate-limits-and-retries-in-python-a-guide-to-polite-scraping-1udxe","title":"Handling Rate Limits and Retries in Python: A Guide to Polite Scraping","subtitle":"Stop hammering servers. Build a scraper that slows itself down, obeys Retry-After, backs off with jitter and knows when to give up.","tldr":"When a site says “Too Many Requests”, a careless retry loop only makes things worse. This guide builds a client that paces itself, waits exactly as long as it’s told, spreads out its retries and stops when it should, with every snippet tested on a deliberately flaky server.","keywords":["Web Scraping","Python","Retries","Rate Limiting","http","Web Development"],"entities":["Bhavya Arora","Web Scraping","Python","Retries","Rate Limiting","http","Web Development","ZyVOP"],"keyTakeaways":["Sooner or later your scraper meets a 429 Too Many Requests.","What it does next decides whether it is a polite client or a nuisance.","A careless script retries instantly, in a tight loop, and makes the problem worse."],"headings":["How Servers Say \"Slow Down\"","A Server to Practice On","Step 1: Don't Trigger the Limit","Step 2: Read Retry-After Correctly","Step 3: Back Off with Jitter","Step 4: Put It Together","The Built-in Option: urllib3's Retry","The Same Idea in httpx","Rate Limits and Retries in Scrapy","Being a Good Citizen","Troubleshooting","Where to Go from Here"],"outboundLinks":[],"contentText":"Sooner or later your scraper meets a 429 Too Many Requests. What it does next decides whether it is a polite client or a nuisance. A careless script retries instantly, in a tight loop, and makes the problem worse. A careful one slows down, listens to what the server says, and gives up gracefully when asked to. This guide builds that careful behavior in four steps: don't trigger limits, read what the server tells you, wait the right amount, and know when to stop. You'll write small, tested helpers for requests, then see the same ideas in httpx and Scrapy. One rule comes first. When a site pushes back, the right response is to slow down. Rotating proxies or user agents to get around a limit is not a fix, because it is the same traffic from more disguises (see our guides on both for the proper uses). Version note: tested with Python 3.13, requests 2.34, urllib3 2.8, httpx 0.28 and Scrapy 2.20. How Servers Say \"Slow Down\" Signal What it usually means What to do 429 Too Many Requests You are sending requests too fast Wait, then retry more slowly 503 Service Unavailable Overloaded or in maintenance Wait, then retry 500, 502, 504 Temporary server or gateway trouble Retry a few times with backoff Timeouts, connection resets Network or server hiccup Retry a few times with backoff 403 Forbidden Usually \"you may not\", not \"slower\" Don't hammer it; read the response and the site's rules 400, 401, 404 Retrying won't change the answer Don't retry The most useful signal is the Retry-After header, defined in the HTTP standard. It comes in two forms: Retry-After: 120 Retry-After: Wed, 21 Oct 2015 07:28:00 GMTThe first is a number of seconds to wait; the second is a date after which you may try again. Your code must handle both. Many sites also report their limits in other headers, such as X-RateLimit-Remaining. The names vary from site to site. The IETF has a draft to standardize RateLimit and RateLimit-Policy fields, but as of November 2026 it is still an Internet-Draft (revision 11, from May 2026), so check which headers your target actually sends. Retry-After is the one you can rely on. A Server to Practice On To test retry logic you need a server that misbehaves on demand. Save this as flaky_server.py: import time from email.utils import formatdate from http.server import BaseHTTPRequestHandler, HTTPServer hits = {} class Flaky(BaseHTTPRequestHandler): def do_GET(self): n = hits[self.path] = hits.get(self.path, 0) + 1 if self.path == \"/limited\" and n &lt;= 2: # too many requests, wait 2 s self.reply(429, \"slow down\", {\"Retry-After\": \"2\"}) elif self.path == \"/maintenance\" and n == 1: # busy, back in about 3 s when = formatdate(time.time() + 3, usegmt=True) self.reply(503, \"back soon\", {\"Retry-After\": when}) else: self.reply(200, f\"ok (hit {n})\") def reply(self, status, text, headers=None): body = text.encode() self.send_response(status) for name, value in (headers or {}).items(): self.send_header(name, value) self.send_header(\"Content-Length\", str(len(body))) self.end_headers() self.wfile.write(body) def log_message(self, *args): pass HTTPServer((\"127.0.0.1\", 8000), Flaky).serve_forever()Run it with python flaky_server.py and install the libraries in another terminal (pip install requests httpx). Paths start working again after a couple of failures, so restart the server between experiments. Step 1: Don't Trigger the Limit The best retry is the one you never need. Instead of sending requests as fast as your code can loop, enforce a minimum gap per site. Better still, make the gap adaptive: widen it after pushback and narrow it gradually after success. Create backoff.py with the shared pieces first: import email.utils import random import time from datetime import datetime, timezone from urllib.parse import urlsplit RETRY_STATUS = {429, 500, 502, 503, 504} MAX_WAIT = 120 # seconds: never sleep longer than this class GaveUp(Exception): \"\"\"Raised when a request can't be completed politely.\"\"\" def retry_after_seconds(value, now=None): \"\"\"Turn a Retry-After header (seconds or HTTP date) into seconds, or None.\"\"\" if not value: return None value = value.strip() if value.isascii() and value.isdigit(): return float(value) try: when = email.utils.parsedate_to_datetime(value) except (TypeError, ValueError): return None if when.tzinfo is None: when = when.replace(tzinfo=timezone.utc) now = now or datetime.now(timezone.utc) return max(0.0, (when - now).total_seconds()) def full_jitter(attempt, base=1.0, cap=60.0): \"\"\"Wait a random time between 0 and an exponentially growing ceiling.\"\"\" return random.uniform(0, min(cap, base * 2 ** attempt))Then add the limiter to the same file: class AdaptiveLimiter: \"\"\"Keeps a minimum gap between requests to each host. The gap doubles after pushback and shrinks slowly after successes. \"\"\" def __init__(self, min_gap=1.0, max_gap=30.0): self.min_gap, self.max_gap = min_gap, max_gap self.gap = {} # host -&gt; current gap in seconds self.next_at = {} # host -&gt; earliest time.monotonic() for the next request def wait(self, url): host = urlsplit(url).netloc now = time.monotonic() start = max(now, self.next_at.get(host, 0.0)) gap = self.gap.get(host, self.min_gap) self.next_at[host] = start + gap * random.uniform(1.0, 1.5) if start &gt; now: time.sleep(start - now) def slow_down(self, url, pause=0.0): host = urlsplit(url).netloc self.gap[host] = min(self.max_gap, self.gap.get(host, self.min_gap) * 2) self.next_at[host] = max(self.next_at.get(host, 0.0), time.monotonic() + pause) def speed_up(self, url): host = urlsplit(url).netloc self.gap[host] = max(self.min_gap, self.gap.get(host, self.min_gap) * 0.9)Call limiter.wait(url) before every request. The random 1.0 to 1.5 multiplier keeps your timing from looking mechanical, and speed_up recovers only 10% per success, so after trouble the scraper stays cautious for a while. The limiter isn't thread-safe, so give each thread its own or guard it with a lock. Step 2: Read Retry-After Correctly retry_after_seconds() above handles both header formats and returns None for junk. A few quick checks show what it does: from datetime import datetime, timezone from backoff import retry_after_seconds assert retry_after_seconds(\"120\") == 120 assert retry_after_seconds(\"soon\") is None assert retry_after_seconds(\"Wed, 21 Oct 2015 07:28:00 GMT\") == 0 # a date in the past now = datetime(2015, 10, 21, 7, 27, 0, tzinfo=timezone.utc) assert retry_after_seconds(\"Wed, 21 Oct 2015 07:28:00 GMT\", now=now) == 60Treat the value as a minimum. Waiting a little longer is always fine; waiting less is not. Step 3: Back Off with Jitter When there is no Retry-After (a 500, a timeout), you have to guess a sensible wait. Exponential backoff doubles the wait after each failure: 1 second, then 2, 4, 8, and so on up to a cap. But if many clients fail at the same moment and all use the same schedule, they all come back at the same moment too, and the server is hit by a wave again and again. Jitter fixes that by randomizing the wait. The AWS Architecture Blog compared the common strategies, and plain backoff without jitter came out worst. The simplest good option is full jitter, which is what full_jitter() above does: wait a random time between 0 and min(cap, base * 2 ** attempt). With the defaults, the ceiling for each attempt is 1, 2, 4, 8, 16, 32 and then 60 seconds, and the actual wait is anywhere below it. Step 4: Put It Together Now combine the pieces in polite.py: import logging import random import time import requests from backoff import (MAX_WAIT, RETRY_STATUS, AdaptiveLimiter, GaveUp, full_jitter, retry_after_seconds) log = logging.getLogger(\"polite\") def polite_get(session, url, limiter, max_attempts=5, **kwargs): \"\"\"GET with rate limiting, Retry-After support and jittered backoff.\"\"\" kwargs.setdefault(\"timeout\", 10) reason = \"no response\" for attempt in range(max_attempts): limiter.wait(url) try: response = session.get(url, **kwargs) except (requests.ConnectionError, requests.Timeout) as exc: reason = type(exc).__name__ pause = full_jitter(attempt) else: if response.status_code not in RETRY_STATUS: limiter.speed_up(url) return response reason = f\"HTTP {response.status_code}\" pause = retry_after_seconds(response.headers.get(\"Retry-After\")) if pause is None: pause = full_jitter(attempt) elif pause &gt; MAX_WAIT: raise GaveUp(f\"{url}: the server asked us to wait {pause:.0f}s\") else: pause += random.uniform(0, 1) # obey it, plus a little jitter log.info(\"%s failed (%s), pausing %.1fs\", url, reason, pause) limiter.slow_down(url, pause) raise GaveUp(f\"{url}: still failing after {max_attempts} attempts ({reason})\") if __name__ == \"__main__\": logging.basicConfig(level=logging.INFO, format=\" %(message)s\") limiter = AdaptiveLimiter(min_gap=0.5) start = time.monotonic() with requests.Session() as session: for path in (\"/limited\", \"/maintenance\", \"/ok\"): response = polite_get(session, f\"http://127.0.0.1:8000{path}\", limiter) elapsed = time.monotonic() - start print(f\"{elapsed:5.1f}s {path:&lt;13} -&gt; {response.status_code} {response.text}\")Run python polite.py against the flaky server: http://127.0.0.1:8000/limited failed (HTTP 429), pausing 2.6s http://127.0.0.1:8000/limited failed (HTTP 429), pausing 2.5s 5.1s /limited -&gt; 200 ok (hit 3) http://127.0.0.1:8000/maintenance failed (HTTP 503), pausing 3.0s 10.8s /maintenance -&gt; 200 ok (hit 2) 15.6s /ok -&gt; 200 ok (hit 1)Your timings will differ slightly because of the jitter. Notice a few things: The pauses obey the server. The first two are Retry-After: 2 plus up to a second of jitter, and the third comes from an HTTP date. Pushback has a memory. /ok waited almost five seconds after the previous success, because the limiter had widened its gap after three failures. It stops when asked to. A Retry-After longer than MAX_WAIT raises GaveUp instead of sleeping for an hour, and so does running out of attempts. The caller decides whether to skip the URL or stop the crawl. It only retries what can succeed. A 404 comes straight back and is not retried. The Built-in Option: urllib3's Retry If you'd rather not write a loop, requests can retry through urllib3's Retry class: import requests from requests.adapters import HTTPAdapter from urllib3.util import Retry retry = Retry( total=5, status_forcelist=[429, 500, 502, 503, 504], backoff_factor=1, # waits 0 s, 2 s, 4 s, 8 s ... between retries backoff_max=60, backoff_jitter=0.5, # adds 0 to 0.5 s on top of each wait respect_retry_after_header=True, retry_after_max=120, # the default is 21,600 seconds (six hours) raise_on_status=False, # return the last response instead of raising ) session = requests.Session() session.mount(\"https://\", HTTPAdapter(max_retries=retry)) session.mount(\"http://\", HTTPAdapter(max_retries=retry))I measured how it behaves against the flaky server, and some defaults will surprise you: Retry-After is honored, in both seconds and date form, for 429 and 503 once those codes are in status_forcelist. The cap is six hours. Without retry_after_max, a server that says \"come back in a day\" can freeze your scraper for hours. The first retry is immediate. With backoff_factor=1 the waits were 0, 2, 4 and 8 seconds. backoff_jitter is additive. It adds a small random amount to the wait, which is not the full jitter described above. Only idempotent methods are retried. POST is excluded by default. Giving up raises. When retries run out, you get a RetryError unless you set raise_on_status=False. It is concise and widely used, but it has no per-site limiter and no \"stop when asked\" rule, so polite_get() above gives you more control. The Same Idea in httpx httpx.HTTPTransport(retries=3) sounds like what you want, but it only retries failed connection attempts. I confirmed that a 500 response is not retried at all. For status codes you need your own loop, and the helpers from backoff.py slot straight in: import time import httpx from backoff import (MAX_WAIT, RETRY_STATUS, GaveUp, full_jitter, retry_after_seconds) def get_with_retries(client, url, limiter, max_attempts=5): for attempt in range(max_attempts): limiter.wait(url) try: response = client.get(url) except httpx.TransportError: pause = full_jitter(attempt) else: if response.status_code not in RETRY_STATUS: limiter.speed_up(url) return response pause = retry_after_seconds(response.headers.get(\"Retry-After\")) if pause is None: pause = full_jitter(attempt) elif pause &gt; MAX_WAIT: raise GaveUp(f\"{url}: the server asked us to wait {pause:.0f}s\") limiter.slow_down(url, pause) raise GaveUp(f\"{url}: still failing after {max_attempts} attempts\")Use it with with httpx.Client(timeout=10) as client: and the same AdaptiveLimiter. Rate Limits and Retries in Scrapy Scrapy has retries built in: failed requests are retried up to RETRY_TIMES (default 2) for the status codes in RETRY_HTTP_CODES (default 408, 429, 500, 502, 503, 504, 522, 524), and a retried request gets a lower priority, so it waits behind the others. AutoThrottle, which we covered in our Scrapy guide, also slows the crawl when the server slows down. What Scrapy doesn't do is read Retry-After. In my test, a 429 that asked for four seconds was followed by the next request within about 1.5 seconds, and the crawl carried on. A small middleware fixes that. It remembers when each site may be contacted again and holds back new requests until then. Put backoff.py inside your project package and add this to myproject/middlewares.py: import asyncio import logging import time from urllib.parse import urlsplit from scrapy.downloadermiddlewares.retry import RetryMiddleware from myproject.backoff import MAX_WAIT, retry_after_seconds log = logging.getLogger(__name__) class RetryAfterMiddleware(RetryMiddleware): \"\"\"Retry as usual, but honor Retry-After for every request to that site.\"\"\" def __init__(self, settings): super().__init__(settings) self.resume_at = {} # host -&gt; time.monotonic() when requests may resume async def process_request(self, request): host = urlsplit(request.url).netloc wait = self.resume_at.get(host, 0) - time.monotonic() if wait &gt; 0: await asyncio.sleep(wait) def process_response(self, request, response): if response.status in self.retry_http_codes: header = response.headers.get(\"Retry-After\", b\"\").decode(\"latin-1\") wait = retry_after_seconds(header) if wait and wait &gt; MAX_WAIT: log.error(\"Retry-After of %.0fs is too long; stopping the crawl\", wait) self._stop_task = asyncio.create_task(self.crawler.stop_async()) return response if wait: host = urlsplit(request.url).netloc resume = time.monotonic() + wait self.resume_at[host] = max(self.resume_at.get(host, 0), resume) return super().process_response(request, response)Then enable it in settings.py: RETRY_TIMES = 4 CONCURRENT_REQUESTS = 1 DOWNLOADER_MIDDLEWARES = { \"scrapy.downloadermiddlewares.retry.RetryMiddleware\": None, # turn off the built-in one \"myproject.middlewares.RetryAfterMiddleware\": 550, # same priority, new behavior }With this in place, the same test paused for the full four seconds, and a Retry-After of 1,000 seconds stopped the crawl cleanly instead of waiting. Two details matter: CONCURRENT_REQUESTS = 1 is deliberate. The middleware can only hold back requests that haven't entered the downloader yet. With Scrapy's default of 16, several requests are already queued and slip through. One request at a time keeps the pause honest. The new project template's limit of 1 request per domain isn't enough on its own, because the queued requests have already passed the middleware. Use stop_async(), not stop(). In Scrapy 2.20, Crawler.stop() is deprecated. Being a Good Citizen Reduce the load you create. Don't fetch pages you already have. Start gentle. One request per second or two is a reasonable opening pace for a small site, and five attempts with a two-minute cap is a reasonable retry budget. These are starting points, not rules, so adjust them to what the site tells you. Check for an API. Official APIs publish their limits and are meant to be called. Our guide \"Is Web Scraping Legal?\" covers the wider picture of terms of service and good practice. Troubleshooting Symptom Likely cause What to try Still getting 429 despite retries Your base rate is too high Raise min_gap or lower concurrency Scraper froze for hours A huge Retry-After with no cap Set MAX_WAIT, or retry_after_max in urllib3 Everything retries at the same instant No jitter Use full_jitter() Scrapy ignores Retry-After The built-in retry middleware doesn't read it Use RetryAfterMiddleware with CONCURRENT_REQUESTS = 1 Retries never succeed A permanent failure (403, 404) is in your retry list Remove it; don't retry those 200 responses full of \"access denied\" Some sites block with a normal status Check the page content, not just the status code Crawl stays slow after one 429 The limiter recovers by 10% per success Expected; it speeds up again over time Where to Go from Here You now have a client that slows itself down, obeys the server, backs off with jitter and knows when to quit. To build on it: Run it on a schedule so it works at quiet times. Cache responses with ETag or Last-Modified so unchanged pages cost nothing. Happy scraping, and please do it politely! Found this helpful? Share it with a fellow developer or leave your questions in the comments.","contentHash":"sha256:46e723482a97c192df38d062e0d2f30723c944675b98ae64649c6023202f6250","authorName":"Bhavya Arora","authorUrl":"https://zyvop.com/author/bhavya","authorSameAs":[],"category":"Web Development","tags":["Web Scraping","Python","Retries","Rate Limiting","http"],"audience":"Software engineers and developers building applications with Web Development","tone":"Instructional, practical, code-first","readingTimeMinutes":10,"wordCount":2644,"faqs":null,"primaryTopic":"Web Development","publishedAt":"2026-11-19T07:24:36.762Z","updatedAt":"2026-11-19T07:24:36.762Z","canonicalUrl":"https://zyvop.com/handling-rate-limits-and-retries-in-python-a-guide-to-polite-scraping-1udxe"}