--- title: "How to Rotate Proxies in Python: A Practical Guide" author: "Bhavya Arora" published: "2026-11-13T06:25:00.026Z" updated: "2026-11-12T06:25:13.547Z" url: "https://zyvop.com/how-to-rotate-proxies-in-python-a-practical-guide-y9ccf" canonical: "https://zyvop.com/how-to-rotate-proxies-in-python-a-practical-guide-y9ccf" tags: ["Web Scraping","Python","aiohttp","Requests","proxies"] --- # How to Rotate Proxies in Python: A Practical Guide Learn how to rotate proxies in Python with requests and aiohttp, from simple random choice to a reusable rotator that retries and benches bad proxies. ![Cover image]() Sooner or later, every serious scraping or data-collection project runs into the same problem: a single proxy (or your own IP address) isn't enough. It goes down, it gets slow, or you simply need to see a site from several locations. The answer is **proxy rotation**: spreading your requests across a pool of proxies instead of relying on just one. In this guide you'll learn how to rotate proxies in Python, starting with a few lines of `requests` code and building up to a reusable rotator class that retries failed requests, benches broken proxies, and works with async code too. > **Heads-up:** all example proxy addresses below (`203.0.113.x`) come from a range reserved for documentation. Replace them with proxies you actually have access to. * * * ## Why Rotate Proxies? There are plenty of legitimate reasons to use a pool instead of a single proxy: - **Reliability.** If one proxy dies, your job keeps running through the others. - **Geo-testing.** Check how a page, price, or ad renders from different countries. - **Spreading permitted workloads.** Large, authorized data collection can be distributed across many exit points instead of funneling through one. - **Privacy.** Avoid tying all your traffic to a single address. ## Use Proxies Responsibly Rotation is a reliability tool, not a license to ignore a site's rules. Keep these principles in mind: - **Follow the site's Terms of Service and** `robots.txt`**.** Rotating your IP doesn't make prohibited scraping permitted. - **Treat "slow down" as an instruction.** If a site rate-limits you (HTTP 429), the right response is to back off, not to hit it from a fresh IP. The code in this post honors `Retry-After` for exactly that reason. - **Don't rotate your way past a block.** A persistent `403 Forbidden` means the site doesn't want your traffic. Look for an official API or ask for permission. - **Never use proxies to access accounts, data, or systems you aren't authorized to use.** ## Proxy Basics ### Types of proxies | Type | Strengths | Trade-offs | | --- | --- | --- | | **Datacenter** | Fast and inexpensive | Easy for sites to identify as non-residential | | **Residential** | IPs belong to real home connections | Slower and pricier | | **Mobile** | Carrier-grade IPs | Usually the most expensive | | **Free public lists** | No cost | Unreliable, slow, and potentially unsafe (see below) | > **A word of caution on free proxies:** whoever runs a proxy can see (and potentially tamper with) any unencrypted traffic passing through it, and can log metadata even for HTTPS. Never send passwords, API keys, or personal data through a proxy you don't trust. ### Proxy URL format Python libraries describe a proxy with a single URL: ```text scheme://username:password@host:port ``` The scheme is usually `http` (works for both HTTP and HTTPS sites via tunneling) or `socks5`. Authentication is optional. ## Using a Single Proxy with `requests` Install the library first: ```bash pip install requests ``` Then pass a `proxies` dictionary to any request: ```python import requests proxy_url = "http://user:pass@203.0.113.10:8080" proxies = {"http": proxy_url, "https": proxy_url} response = requests.get("https://httpbin.org/ip", proxies=proxies, timeout=10) print(response.json()) # {'origin': '203.0.113.10'} ``` The [httpbin.org/ip]() endpoint echoes the IP address it sees, which makes it handy for confirming that your traffic really goes through the proxy. A few details worth knowing: - Both dictionary keys usually point to the same `http://` proxy URL. The key says which _target_ scheme the proxy is used for; the proxy URL's own scheme describes how you talk to the _proxy_. - `requests` also reads the `HTTP_PROXY` and `HTTPS_PROXY` environment variables, but a `proxies=` argument you pass explicitly takes precedence. - Always set a `timeout`. Dead proxies often hang rather than fail fast. ### Special characters in credentials If your username or password contains characters like `@`, `:` or `/`, URL-encode them. Otherwise the proxy URL is parsed incorrectly and `requests` typically fails with an `InvalidURL` error: ```python from urllib.parse import quote username = quote("my.user@example.com", safe="") password = quote("p@ss:word/1", safe="") proxy_url = f"http://{username}:{password}@203.0.113.10:8080" print(proxy_url) ``` Better still, don't hardcode credentials at all. Load them from environment variables or a secrets manager. ### SOCKS proxies For SOCKS support, install the extra and change the scheme: ```bash pip install "requests[socks]" ``` ```python proxy_url = "socks5h://user:pass@203.0.113.10:1080" ``` The `socks5h` variant resolves DNS through the proxy as well, which avoids leaking your lookups to your own resolver. ## Rotation Strategies ### 1\. Random choice The simplest approach: pick any proxy for each request. ```python import itertools import random PROXIES = [ "http://user:pass@203.0.113.10:8080", "http://user:pass@203.0.113.11:8080", "http://user:pass@203.0.113.12:8080", ] def random_proxy(): return random.choice(PROXIES) ``` It's easy, but you can't guarantee an even spread, and the same proxy may be chosen several times in a row. ### 2\. Round robin `itertools.cycle` walks through the list in order and loops forever, so every proxy gets an equal share: ```python proxy_cycle = itertools.cycle(PROXIES) def next_proxy(): return next(proxy_cycle) ``` ### 3\. Let the provider rotate for you Many paid services expose a single "gateway" endpoint that assigns a different exit IP behind the scenes. In that case your code needs no rotation logic at all: ```python gateway = "http://user:pass@gateway.example-proxy.com:8000" proxies = {"http": gateway, "https": gateway} ``` Check your provider's documentation. Some rotate on every request, others keep a "sticky" IP for a few minutes, which is useful when a multi-step flow (like paging through results) should come from one address. ### Which should you use? | Strategy | Best for | Weakness | | --- | --- | --- | | Random | Quick scripts | Uneven distribution | | Round robin | Fair, predictable spread | Doesn't notice dead proxies | | Provider gateway | Large pools with zero maintenance | Costs money, less control | | **Rotator with health tracking** | Anything you'll run repeatedly | A bit more code (below) | ## Building a Reusable Rotator Random and round-robin both share a flaw: they keep handing out a proxy even after it has stopped working. A better rotator tracks failures and temporarily benches proxies that misbehave. ```python import time from dataclasses import dataclass from urllib.parse import urlsplit @dataclass class Proxy: url: str failures: int = 0 banned_until: float = 0.0 @property def label(self) -> str: """Host and port only, so credentials never end up in logs.""" return urlsplit(self.url).netloc.rpartition("@")[2] class ProxyRotator: def __init__(self, proxy_urls, max_failures=3, cooldown=300): self.proxies = [Proxy(url) for url in proxy_urls] self.max_failures = max_failures # consecutive failures before benching self.cooldown = cooldown # seconds a benched proxy sits out self._counter = 0 def get(self) -> Proxy: """Return the next healthy proxy in round-robin order.""" now = time.time() healthy = [p for p in self.proxies if p.banned_until <= now] if not healthy: raise RuntimeError("No healthy proxies left") proxy = healthy[self._counter % len(healthy)] self._counter += 1 return proxy def report_success(self, proxy: Proxy) -> None: proxy.failures = 0 def report_failure(self, proxy: Proxy) -> None: proxy.failures += 1 if proxy.failures >= self.max_failures: proxy.banned_until = time.time() + self.cooldown proxy.failures = 0 ``` What this gives you: - **Round-robin rotation** over whichever proxies are currently healthy. - **Automatic benching.** After `max_failures` failures in a row, a proxy sits out for `cooldown` seconds, then rejoins the pool. - **Safe logging.** The `label` property prints only host and port, never your username and password. > This class isn't thread-safe. If you share one rotator across threads, wrap `get()`, `report_success()` and `report_failure()` in a `threading.Lock`. ## Fetching with Automatic Failover Now pair the rotator with a request function that switches proxies when one fails: ```python import random import time import requests HEADERS = {"User-Agent": "MyLearningScraper/1.0 (educational project)"} PROXY_ERROR_STATUS = {407, 502, 504} # what a proxy typically returns when it fails SITE_BUSY_STATUS = {429, 500, 503} # the site itself is asking you to back off def fetch(url, rotator, max_attempts=5, timeout=10): """GET a URL, switching to a different proxy whenever one fails.""" last_error = None for attempt in range(1, max_attempts + 1): proxy = rotator.get() try: response = requests.get( url, headers=HEADERS, proxies={"http": proxy.url, "https": proxy.url}, timeout=timeout, ) except requests.RequestException as exc: # ProxyError, timeouts, SSL errors... last_error = exc rotator.report_failure(proxy) print(f"Attempt {attempt}: {proxy.label} failed ({type(exc).__name__})") time.sleep(random.uniform(0.5, 1.5)) continue if response.status_code in SITE_BUSY_STATUS: # the site says "slow down": respect it retry_after = response.headers.get("Retry-After", "") delay = int(retry_after) if retry_after.isdigit() else 2 ** attempt print(f"Attempt {attempt}: site returned {response.status_code}, waiting {delay}s") time.sleep(min(delay, 60)) continue if response.status_code in PROXY_ERROR_STATUS: # bad proxy login, or proxy can't reach the site rotator.report_failure(proxy) print(f"Attempt {attempt}: {proxy.label} returned {response.status_code}") time.sleep(random.uniform(0.5, 1.5)) continue rotator.report_success(proxy) response.raise_for_status() # 403, 404...: don't rotate, just raise return response raise RuntimeError(f"All {max_attempts} attempts failed") from last_error ``` The key decisions here: - **Network-level errors** (connection refused, timeouts, TLS failures) count against the proxy and trigger an immediate switch. - **407, 502 and 504 responses** are what proxies typically return when your login is wrong or they can't reach the site, so they count against the proxy. - **429, 500 and 503 come from the _site_,** which is asking you to slow down or is having trouble. The code waits (honoring `Retry-After` when provided) and doesn't blame the proxy, so a site outage can't bench your whole pool. - **Other 4xx errors** such as 403 or 404 are raised straight away. Retrying them through a different proxy would just be hammering a site that already said no. ## Plugging It into a BeautifulSoup Scraper If you've built a scraper with BeautifulSoup, swapping in rotation is a one-line change: replace your `requests.get` call with `fetch`. ```python from bs4 import BeautifulSoup PROXY_URLS = [ "http://user:pass@203.0.113.10:8080", "http://user:pass@203.0.113.11:8080", "http://user:pass@203.0.113.12:8080", ] rotator = ProxyRotator(PROXY_URLS) response = fetch("https://books.toscrape.com/", rotator) soup = BeautifulSoup(response.content, "html.parser") print(soup.title.get_text(strip=True)) ``` Because the rotator keeps its state between calls, you can loop over hundreds of pages and the load will spread evenly across your healthy proxies. ## Filtering Out Dead Proxies Before You Start Proxy lists, especially free ones, are full of dead entries. It's cheaper to weed them out up front than to burn retries on them mid-crawl. ```python from concurrent.futures import ThreadPoolExecutor import requests def load_proxies(path): """Read one proxy URL per line, ignoring blank lines and # comments.""" with open(path, encoding="utf-8") as f: lines = (line.strip() for line in f) return [line for line in lines if line and not line.startswith("#")] def check_proxy(proxy_url, timeout=8): """Return the exit IP if the proxy works, otherwise None.""" try: response = requests.get( "https://httpbin.org/ip", proxies={"http": proxy_url, "https": proxy_url}, timeout=timeout, ) response.raise_for_status() return response.json()["origin"] except (requests.RequestException, ValueError, KeyError): return None def healthy_proxies(candidates, workers=10): """Test proxies in parallel and keep only the ones that respond.""" with ThreadPoolExecutor(max_workers=workers) as pool: exit_ips = list(pool.map(check_proxy, candidates)) return [url for url, ip in zip(candidates, exit_ips) if ip] ``` Use it like this: ```python candidates = load_proxies("proxies.txt") rotator = ProxyRotator(healthy_proxies(candidates)) ``` ## Rotating Proxies Asynchronously When you need many requests in flight at once, `aiohttp` accepts a `proxy` argument on each request, which makes rotation straightforward: ```bash pip install aiohttp ``` ```python import asyncio import itertools import aiohttp PROXIES = [ "http://user:pass@203.0.113.10:8080", "http://user:pass@203.0.113.11:8080", "http://user:pass@203.0.113.12:8080", ] proxy_cycle = itertools.cycle(PROXIES) async def fetch(session, url, proxy, semaphore): async with semaphore: # cap how many requests run at once try: timeout = aiohttp.ClientTimeout(total=10) async with session.get(url, proxy=proxy, timeout=timeout) as resp: resp.raise_for_status() return url, resp.status, await resp.text() except (aiohttp.ClientError, asyncio.TimeoutError) as exc: return url, None, type(exc).__name__ async def main(urls): semaphore = asyncio.Semaphore(5) async with aiohttp.ClientSession() as session: tasks = [fetch(session, url, next(proxy_cycle), semaphore) for url in urls] return await asyncio.gather(*tasks) if __name__ == "__main__": results = asyncio.run(main(["https://httpbin.org/ip"] * 6)) for url, status, body in results: print(status, body[:60]) ``` Concurrency multiplies your request rate, so keep the semaphore small and stay within what the target site allows. ### A quick note on `httpx` If you prefer `httpx`, create a client with the `proxy` argument: ```python import httpx with httpx.Client(proxy=random_proxy(), timeout=10) as client: print(client.get("https://httpbin.org/ip").json()) ``` Older tutorials use a `proxies=` argument, which was removed in `httpx` 0.28. Use `proxy=` on current versions. For rotation, build a new client per proxy. ## Best Practices - **Add delays and jitter.** Pause between requests, ideally with a random component, so you don't hit a site in lockstep. - **Rotate for reliability, not to dodge limits.** Honor `429` and `Retry-After`, and stop if you keep getting `403`. - **Always set timeouts.** A proxy that hangs is worse than one that fails. - **Keep sessions consistent.** If a site uses cookies, create one `requests.Session` per proxy so each exit IP carries a coherent set of cookies. - **Monitor your pool.** Log successes and failures per proxy (by `label`, never with credentials) and replace the chronic offenders. - **Protect credentials.** Keep them out of source control and logs. - **Prefer HTTPS targets.** TLS keeps the content of your traffic private from the proxy operator. ## Troubleshooting | Symptom | Likely cause | What to try | | --- | --- | --- | | `ProxyError` / connection refused | Proxy is down, or wrong host or port | Test it with `check_proxy`; verify the address | | `407 Proxy Authentication Required` | Wrong credentials, or special characters not URL-encoded | Re-check them and use `urllib.parse.quote` | | `ConnectTimeout` / `ReadTimeout` | Proxy is slow or overloaded | Lower the timeout and let the rotator bench it | | `SSLError` | Proxy intercepts TLS, or the wrong proxy scheme is used | Use `http://` for an HTTP proxy URL; avoid proxies that tamper with certificates | | Frequent `429` | You're sending requests too fast | Increase delays and respect `Retry-After` | | Persistent `403` | The site is refusing your traffic | Review its terms; look for an API or ask permission | ## Wrapping Up Proxy rotation in Python comes down to three ideas: describe each proxy with a URL, pick the next one with a sensible strategy (random, round robin, or a provider gateway), and handle failures so a single bad proxy never sinks your run. The `ProxyRotator` and `fetch` function above give you a solid foundation: round-robin rotation, automatic failover, benching with a cooldown, and respect for rate limits. From here, you could extend the rotator with weighted selection (favoring faster proxies), per-proxy statistics, or a lock for multithreaded use. Happy scraping, and please do it politely! * * * _Found this helpful? Share it with a fellow developer or leave your questions in the comments._