{"schemaVersion":"1.0","type":"TechArticle","types":["Article","TechArticle"],"slug":"how-to-rotate-proxies-in-python-a-practical-guide-y9ccf","url":"https://zyvop.com/how-to-rotate-proxies-in-python-a-practical-guide-y9ccf","title":"How to Rotate Proxies in Python: A Practical Guide","subtitle":"Learn how to rotate proxies in Python with requests and aiohttp, from simple random choice to a reusable rotator that retries and benches bad proxies.","tldr":"One proxy isn’t enough for long. It can slow down, stop responding, or drop out halfway through a job. This guide shows how to rotate through a pool of proxies in Python so your scraper keeps running when one fails, with simple code you can copy, adapt, and use in your own projects.","keywords":["Web Scraping","Python","aiohttp","Requests","proxies","Tutorial"],"entities":["Bhavya Arora","Web Scraping","Python","aiohttp","Requests","proxies","Tutorial","ZyVOP"],"keyTakeaways":["Sooner or later, every serious scraping or data-collection project runs into the same problem: a single proxy (or your own IP address) isn't enough.","It goes down, it gets slow, or you simply need to see a site from several locations.","The answer is proxy rotation: spreading your requests across a pool of proxies instead of relying on just one."],"headings":["Why Rotate Proxies?","Use Proxies Responsibly","Proxy Basics","Types of proxies","Proxy URL format","Using a Single Proxy with requests","Special characters in credentials","SOCKS proxies","Rotation Strategies","1. Random choice","2. Round robin","3. Let the provider rotate for you","Which should you use?","Building a Reusable Rotator","Fetching with Automatic Failover","Plugging It into a BeautifulSoup Scraper","Filtering Out Dead Proxies Before You Start","Rotating Proxies Asynchronously","A quick note on httpx","Best Practices","Troubleshooting","Wrapping Up"],"outboundLinks":["https://httpbin.org/ip"],"contentText":"Sooner or later, every serious scraping or data-collection project runs into the same problem: a single proxy (or your own IP address) isn't enough. It goes down, it gets slow, or you simply need to see a site from several locations. The answer is proxy rotation: spreading your requests across a pool of proxies instead of relying on just one. In this guide you'll learn how to rotate proxies in Python, starting with a few lines of requests code and building up to a reusable rotator class that retries failed requests, benches broken proxies, and works with async code too. Heads-up: all example proxy addresses below (203.0.113.x) come from a range reserved for documentation. Replace them with proxies you actually have access to. Why Rotate Proxies? There are plenty of legitimate reasons to use a pool instead of a single proxy: Reliability. If one proxy dies, your job keeps running through the others. Geo-testing. Check how a page, price, or ad renders from different countries. Spreading permitted workloads. Large, authorized data collection can be distributed across many exit points instead of funneling through one. Privacy. Avoid tying all your traffic to a single address. Use Proxies Responsibly Rotation is a reliability tool, not a license to ignore a site's rules. Keep these principles in mind: Follow the site's Terms of Service and robots.txt. Rotating your IP doesn't make prohibited scraping permitted. Treat \"slow down\" as an instruction. If a site rate-limits you (HTTP 429), the right response is to back off, not to hit it from a fresh IP. The code in this post honors Retry-After for exactly that reason. Don't rotate your way past a block. A persistent 403 Forbidden means the site doesn't want your traffic. Look for an official API or ask for permission. Never use proxies to access accounts, data, or systems you aren't authorized to use. Proxy Basics Types of proxies Type Strengths Trade-offs Datacenter Fast and inexpensive Easy for sites to identify as non-residential Residential IPs belong to real home connections Slower and pricier Mobile Carrier-grade IPs Usually the most expensive Free public lists No cost Unreliable, slow, and potentially unsafe (see below) A word of caution on free proxies: whoever runs a proxy can see (and potentially tamper with) any unencrypted traffic passing through it, and can log metadata even for HTTPS. Never send passwords, API keys, or personal data through a proxy you don't trust. Proxy URL format Python libraries describe a proxy with a single URL: scheme://username:password@host:portThe scheme is usually http (works for both HTTP and HTTPS sites via tunneling) or socks5. Authentication is optional. Using a Single Proxy with requests Install the library first: pip install requestsThen pass a proxies dictionary to any request: import requests proxy_url = \"http://user:pass@203.0.113.10:8080\" proxies = {\"http\": proxy_url, \"https\": proxy_url} response = requests.get(\"https://httpbin.org/ip\", proxies=proxies, timeout=10) print(response.json()) # {'origin': '203.0.113.10'}The httpbin.org/ip endpoint echoes the IP address it sees, which makes it handy for confirming that your traffic really goes through the proxy. A few details worth knowing: Both dictionary keys usually point to the same http:// proxy URL. The key says which target scheme the proxy is used for; the proxy URL's own scheme describes how you talk to the proxy. requests also reads the HTTP_PROXY and HTTPS_PROXY environment variables, but a proxies= argument you pass explicitly takes precedence. Always set a timeout. Dead proxies often hang rather than fail fast. Special characters in credentials If your username or password contains characters like @, : or /, URL-encode them. Otherwise the proxy URL is parsed incorrectly and requests typically fails with an InvalidURL error: from urllib.parse import quote username = quote(\"my.user@example.com\", safe=\"\") password = quote(\"p@ss:word/1\", safe=\"\") proxy_url = f\"http://{username}:{password}@203.0.113.10:8080\" print(proxy_url)Better still, don't hardcode credentials at all. Load them from environment variables or a secrets manager. SOCKS proxies For SOCKS support, install the extra and change the scheme: pip install \"requests[socks]\"proxy_url = \"socks5h://user:pass@203.0.113.10:1080\"The socks5h variant resolves DNS through the proxy as well, which avoids leaking your lookups to your own resolver. Rotation Strategies 1. Random choice The simplest approach: pick any proxy for each request. import itertools import random PROXIES = [ \"http://user:pass@203.0.113.10:8080\", \"http://user:pass@203.0.113.11:8080\", \"http://user:pass@203.0.113.12:8080\", ] def random_proxy(): return random.choice(PROXIES)It's easy, but you can't guarantee an even spread, and the same proxy may be chosen several times in a row. 2. Round robin itertools.cycle walks through the list in order and loops forever, so every proxy gets an equal share: proxy_cycle = itertools.cycle(PROXIES) def next_proxy(): return next(proxy_cycle)3. Let the provider rotate for you Many paid services expose a single \"gateway\" endpoint that assigns a different exit IP behind the scenes. In that case your code needs no rotation logic at all: gateway = \"http://user:pass@gateway.example-proxy.com:8000\" proxies = {\"http\": gateway, \"https\": gateway}Check your provider's documentation. Some rotate on every request, others keep a \"sticky\" IP for a few minutes, which is useful when a multi-step flow (like paging through results) should come from one address. Which should you use? Strategy Best for Weakness Random Quick scripts Uneven distribution Round robin Fair, predictable spread Doesn't notice dead proxies Provider gateway Large pools with zero maintenance Costs money, less control Rotator with health tracking Anything you'll run repeatedly A bit more code (below) Building a Reusable Rotator Random and round-robin both share a flaw: they keep handing out a proxy even after it has stopped working. A better rotator tracks failures and temporarily benches proxies that misbehave. import time from dataclasses import dataclass from urllib.parse import urlsplit @dataclass class Proxy: url: str failures: int = 0 banned_until: float = 0.0 @property def label(self) -&gt; str: \"\"\"Host and port only, so credentials never end up in logs.\"\"\" return urlsplit(self.url).netloc.rpartition(\"@\")[2] class ProxyRotator: def __init__(self, proxy_urls, max_failures=3, cooldown=300): self.proxies = [Proxy(url) for url in proxy_urls] self.max_failures = max_failures # consecutive failures before benching self.cooldown = cooldown # seconds a benched proxy sits out self._counter = 0 def get(self) -&gt; Proxy: \"\"\"Return the next healthy proxy in round-robin order.\"\"\" now = time.time() healthy = [p for p in self.proxies if p.banned_until &lt;= now] if not healthy: raise RuntimeError(\"No healthy proxies left\") proxy = healthy[self._counter % len(healthy)] self._counter += 1 return proxy def report_success(self, proxy: Proxy) -&gt; None: proxy.failures = 0 def report_failure(self, proxy: Proxy) -&gt; None: proxy.failures += 1 if proxy.failures &gt;= self.max_failures: proxy.banned_until = time.time() + self.cooldown proxy.failures = 0What this gives you: Round-robin rotation over whichever proxies are currently healthy. Automatic benching. After max_failures failures in a row, a proxy sits out for cooldown seconds, then rejoins the pool. Safe logging. The label property prints only host and port, never your username and password. This class isn't thread-safe. If you share one rotator across threads, wrap get(), report_success() and report_failure() in a threading.Lock. Fetching with Automatic Failover Now pair the rotator with a request function that switches proxies when one fails: import random import time import requests HEADERS = {\"User-Agent\": \"MyLearningScraper/1.0 (educational project)\"} PROXY_ERROR_STATUS = {407, 502, 504} # what a proxy typically returns when it fails SITE_BUSY_STATUS = {429, 500, 503} # the site itself is asking you to back off def fetch(url, rotator, max_attempts=5, timeout=10): \"\"\"GET a URL, switching to a different proxy whenever one fails.\"\"\" last_error = None for attempt in range(1, max_attempts + 1): proxy = rotator.get() try: response = requests.get( url, headers=HEADERS, proxies={\"http\": proxy.url, \"https\": proxy.url}, timeout=timeout, ) except requests.RequestException as exc: # ProxyError, timeouts, SSL errors... last_error = exc rotator.report_failure(proxy) print(f\"Attempt {attempt}: {proxy.label} failed ({type(exc).__name__})\") time.sleep(random.uniform(0.5, 1.5)) continue if response.status_code in SITE_BUSY_STATUS: # the site says \"slow down\": respect it retry_after = response.headers.get(\"Retry-After\", \"\") delay = int(retry_after) if retry_after.isdigit() else 2 ** attempt print(f\"Attempt {attempt}: site returned {response.status_code}, waiting {delay}s\") time.sleep(min(delay, 60)) continue if response.status_code in PROXY_ERROR_STATUS: # bad proxy login, or proxy can't reach the site rotator.report_failure(proxy) print(f\"Attempt {attempt}: {proxy.label} returned {response.status_code}\") time.sleep(random.uniform(0.5, 1.5)) continue rotator.report_success(proxy) response.raise_for_status() # 403, 404...: don't rotate, just raise return response raise RuntimeError(f\"All {max_attempts} attempts failed\") from last_errorThe key decisions here: Network-level errors (connection refused, timeouts, TLS failures) count against the proxy and trigger an immediate switch. 407, 502 and 504 responses are what proxies typically return when your login is wrong or they can't reach the site, so they count against the proxy. 429, 500 and 503 come from the site, which is asking you to slow down or is having trouble. The code waits (honoring Retry-After when provided) and doesn't blame the proxy, so a site outage can't bench your whole pool. Other 4xx errors such as 403 or 404 are raised straight away. Retrying them through a different proxy would just be hammering a site that already said no. Plugging It into a BeautifulSoup Scraper If you've built a scraper with BeautifulSoup, swapping in rotation is a one-line change: replace your requests.get call with fetch. from bs4 import BeautifulSoup PROXY_URLS = [ \"http://user:pass@203.0.113.10:8080\", \"http://user:pass@203.0.113.11:8080\", \"http://user:pass@203.0.113.12:8080\", ] rotator = ProxyRotator(PROXY_URLS) response = fetch(\"https://books.toscrape.com/\", rotator) soup = BeautifulSoup(response.content, \"html.parser\") print(soup.title.get_text(strip=True))Because the rotator keeps its state between calls, you can loop over hundreds of pages and the load will spread evenly across your healthy proxies. Filtering Out Dead Proxies Before You Start Proxy lists, especially free ones, are full of dead entries. It's cheaper to weed them out up front than to burn retries on them mid-crawl. from concurrent.futures import ThreadPoolExecutor import requests def load_proxies(path): \"\"\"Read one proxy URL per line, ignoring blank lines and # comments.\"\"\" with open(path, encoding=\"utf-8\") as f: lines = (line.strip() for line in f) return [line for line in lines if line and not line.startswith(\"#\")] def check_proxy(proxy_url, timeout=8): \"\"\"Return the exit IP if the proxy works, otherwise None.\"\"\" try: response = requests.get( \"https://httpbin.org/ip\", proxies={\"http\": proxy_url, \"https\": proxy_url}, timeout=timeout, ) response.raise_for_status() return response.json()[\"origin\"] except (requests.RequestException, ValueError, KeyError): return None def healthy_proxies(candidates, workers=10): \"\"\"Test proxies in parallel and keep only the ones that respond.\"\"\" with ThreadPoolExecutor(max_workers=workers) as pool: exit_ips = list(pool.map(check_proxy, candidates)) return [url for url, ip in zip(candidates, exit_ips) if ip]Use it like this: candidates = load_proxies(\"proxies.txt\") rotator = ProxyRotator(healthy_proxies(candidates))Rotating Proxies Asynchronously When you need many requests in flight at once, aiohttp accepts a proxy argument on each request, which makes rotation straightforward: pip install aiohttpimport asyncio import itertools import aiohttp PROXIES = [ \"http://user:pass@203.0.113.10:8080\", \"http://user:pass@203.0.113.11:8080\", \"http://user:pass@203.0.113.12:8080\", ] proxy_cycle = itertools.cycle(PROXIES) async def fetch(session, url, proxy, semaphore): async with semaphore: # cap how many requests run at once try: timeout = aiohttp.ClientTimeout(total=10) async with session.get(url, proxy=proxy, timeout=timeout) as resp: resp.raise_for_status() return url, resp.status, await resp.text() except (aiohttp.ClientError, asyncio.TimeoutError) as exc: return url, None, type(exc).__name__ async def main(urls): semaphore = asyncio.Semaphore(5) async with aiohttp.ClientSession() as session: tasks = [fetch(session, url, next(proxy_cycle), semaphore) for url in urls] return await asyncio.gather(*tasks) if __name__ == \"__main__\": results = asyncio.run(main([\"https://httpbin.org/ip\"] * 6)) for url, status, body in results: print(status, body[:60])Concurrency multiplies your request rate, so keep the semaphore small and stay within what the target site allows. A quick note on httpx If you prefer httpx, create a client with the proxy argument: import httpx with httpx.Client(proxy=random_proxy(), timeout=10) as client: print(client.get(\"https://httpbin.org/ip\").json())Older tutorials use a proxies= argument, which was removed in httpx 0.28. Use proxy= on current versions. For rotation, build a new client per proxy. Best Practices Add delays and jitter. Pause between requests, ideally with a random component, so you don't hit a site in lockstep. Rotate for reliability, not to dodge limits. Honor 429 and Retry-After, and stop if you keep getting 403. Always set timeouts. A proxy that hangs is worse than one that fails. Keep sessions consistent. If a site uses cookies, create one requests.Session per proxy so each exit IP carries a coherent set of cookies. Monitor your pool. Log successes and failures per proxy (by label, never with credentials) and replace the chronic offenders. Protect credentials. Keep them out of source control and logs. Prefer HTTPS targets. TLS keeps the content of your traffic private from the proxy operator. Troubleshooting Symptom Likely cause What to try ProxyError / connection refused Proxy is down, or wrong host or port Test it with check_proxy; verify the address 407 Proxy Authentication Required Wrong credentials, or special characters not URL-encoded Re-check them and use urllib.parse.quote ConnectTimeout / ReadTimeout Proxy is slow or overloaded Lower the timeout and let the rotator bench it SSLError Proxy intercepts TLS, or the wrong proxy scheme is used Use http:// for an HTTP proxy URL; avoid proxies that tamper with certificates Frequent 429 You're sending requests too fast Increase delays and respect Retry-After Persistent 403 The site is refusing your traffic Review its terms; look for an API or ask permission Wrapping Up Proxy rotation in Python comes down to three ideas: describe each proxy with a URL, pick the next one with a sensible strategy (random, round robin, or a provider gateway), and handle failures so a single bad proxy never sinks your run. The ProxyRotator and fetch function above give you a solid foundation: round-robin rotation, automatic failover, benching with a cooldown, and respect for rate limits. From here, you could extend the rotator with weighted selection (favoring faster proxies), per-proxy statistics, or a lock for multithreaded use. Happy scraping, and please do it politely! Found this helpful? Share it with a fellow developer or leave your questions in the comments.","contentHash":"sha256:0754ce7a8a126de3a50ded806e91f1e191d108d5791fb1e9fe310f16540e37ed","authorName":"Bhavya Arora","authorUrl":"https://zyvop.com/author/bhavya","authorSameAs":[],"category":"Tutorial","tags":["Web Scraping","Python","aiohttp","Requests","proxies"],"audience":"Software engineers and developers building applications with Tutorial","tone":"Instructional, practical, code-first","readingTimeMinutes":9,"wordCount":2350,"faqs":null,"primaryTopic":"Tutorial","publishedAt":"2026-11-13T06:25:00.026Z","updatedAt":"2026-11-12T06:25:13.547Z","canonicalUrl":"https://zyvop.com/how-to-rotate-proxies-in-python-a-practical-guide-y9ccf"}