
Sooner or later, every serious scraping or data-collection project runs into the same problem: a single proxy (or your own IP address) isn't enough. It goes down, it gets slow, or you simply need to see a site from several locations. The answer is proxy rotation: spreading your requests across a pool of proxies instead of relying on just one.
In this guide you'll learn how to rotate proxies in Python, starting with a few lines of requests code and building up to a reusable rotator class that retries failed requests, benches broken proxies, and works with async code too.
Heads-up: all example proxy addresses below (
203.0.113.x) come from a range reserved for documentation. Replace them with proxies you actually have access to.
Why Rotate Proxies?
There are plenty of legitimate reasons to use a pool instead of a single proxy:
Reliability. If one proxy dies, your job keeps running through the others.
Geo-testing. Check how a page, price, or ad renders from different countries.
Spreading permitted workloads. Large, authorized data collection can be distributed across many exit points instead of funneling through one.
Privacy. Avoid tying all your traffic to a single address.
Use Proxies Responsibly
Rotation is a reliability tool, not a license to ignore a site's rules. Keep these principles in mind:
Follow the site's Terms of Service and
robots.txt. Rotating your IP doesn't make prohibited scraping permitted.Treat "slow down" as an instruction. If a site rate-limits you (HTTP 429), the right response is to back off, not to hit it from a fresh IP. The code in this post honors
Retry-Afterfor exactly that reason.Don't rotate your way past a block. A persistent
403 Forbiddenmeans the site doesn't want your traffic. Look for an official API or ask for permission.Never use proxies to access accounts, data, or systems you aren't authorized to use.
Proxy Basics
Types of proxies
Type | Strengths | Trade-offs |
|---|---|---|
Datacenter | Fast and inexpensive | Easy for sites to identify as non-residential |
Residential | IPs belong to real home connections | Slower and pricier |
Mobile | Carrier-grade IPs | Usually the most expensive |
Free public lists | No cost | Unreliable, slow, and potentially unsafe (see below) |
A word of caution on free proxies: whoever runs a proxy can see (and potentially tamper with) any unencrypted traffic passing through it, and can log metadata even for HTTPS. Never send passwords, API keys, or personal data through a proxy you don't trust.
Proxy URL format
Python libraries describe a proxy with a single URL:
scheme://username:password@host:portThe scheme is usually http (works for both HTTP and HTTPS sites via tunneling) or socks5. Authentication is optional.
Using a Single Proxy with requests
Install the library first:
pip install requestsThen pass a proxies dictionary to any request:
import requests
proxy_url = "http://user:[email protected]:8080"
proxies = {"http": proxy_url, "https": proxy_url}
response = requests.get("https://httpbin.org/ip", proxies=proxies, timeout=10)
print(response.json()) # {'origin': '203.0.113.10'}The httpbin.org/ip endpoint echoes the IP address it sees, which makes it handy for confirming that your traffic really goes through the proxy.
A few details worth knowing:
Both dictionary keys usually point to the same
http://proxy URL. The key says which target scheme the proxy is used for; the proxy URL's own scheme describes how you talk to the proxy.requestsalso reads theHTTP_PROXYandHTTPS_PROXYenvironment variables, but aproxies=argument you pass explicitly takes precedence.Always set a
timeout. Dead proxies often hang rather than fail fast.
Special characters in credentials
If your username or password contains characters like @, : or /, URL-encode them. Otherwise the proxy URL is parsed incorrectly and requests typically fails with an InvalidURL error:
from urllib.parse import quote
username = quote("[email protected]", safe="")
password = quote("p@ss:word/1", safe="")
proxy_url = f"http://{username}:{password}@203.0.113.10:8080"
print(proxy_url)Better still, don't hardcode credentials at all. Load them from environment variables or a secrets manager.
SOCKS proxies
For SOCKS support, install the extra and change the scheme:
pip install "requests[socks]"proxy_url = "socks5h://user:[email protected]:1080"The socks5h variant resolves DNS through the proxy as well, which avoids leaking your lookups to your own resolver.
Rotation Strategies
1. Random choice
The simplest approach: pick any proxy for each request.
import itertools
import random
PROXIES = [
"http://user:[email protected]:8080",
"http://user:[email protected]:8080",
"http://user:[email protected]:8080",
]
def random_proxy():
return random.choice(PROXIES)It's easy, but you can't guarantee an even spread, and the same proxy may be chosen several times in a row.
2. Round robin
itertools.cycle walks through the list in order and loops forever, so every proxy gets an equal share:
proxy_cycle = itertools.cycle(PROXIES)
def next_proxy():
return next(proxy_cycle)3. Let the provider rotate for you
Many paid services expose a single "gateway" endpoint that assigns a different exit IP behind the scenes. In that case your code needs no rotation logic at all:
gateway = "http://user:[email protected]:8000"
proxies = {"http": gateway, "https": gateway}Check your provider's documentation. Some rotate on every request, others keep a "sticky" IP for a few minutes, which is useful when a multi-step flow (like paging through results) should come from one address.
Which should you use?
Strategy | Best for | Weakness |
|---|---|---|
Random | Quick scripts | Uneven distribution |
Round robin | Fair, predictable spread | Doesn't notice dead proxies |
Provider gateway | Large pools with zero maintenance | Costs money, less control |
Rotator with health tracking | Anything you'll run repeatedly | A bit more code (below) |
Building a Reusable Rotator
Random and round-robin both share a flaw: they keep handing out a proxy even after it has stopped working. A better rotator tracks failures and temporarily benches proxies that misbehave.
import time
from dataclasses import dataclass
from urllib.parse import urlsplit
@dataclass
class Proxy:
url: str
failures: int = 0
banned_until: float = 0.0
@property
def label(self) -> str:
"""Host and port only, so credentials never end up in logs."""
return urlsplit(self.url).netloc.rpartition("@")[2]
class ProxyRotator:
def __init__(self, proxy_urls, max_failures=3, cooldown=300):
self.proxies = [Proxy(url) for url in proxy_urls]
self.max_failures = max_failures # consecutive failures before benching
self.cooldown = cooldown # seconds a benched proxy sits out
self._counter = 0
def get(self) -> Proxy:
"""Return the next healthy proxy in round-robin order."""
now = time.time()
healthy = [p for p in self.proxies if p.banned_until <= now]
if not healthy:
raise RuntimeError("No healthy proxies left")
proxy = healthy[self._counter % len(healthy)]
self._counter += 1
return proxy
def report_success(self, proxy: Proxy) -> None:
proxy.failures = 0
def report_failure(self, proxy: Proxy) -> None:
proxy.failures += 1
if proxy.failures >= self.max_failures:
proxy.banned_until = time.time() + self.cooldown
proxy.failures = 0What this gives you:
Round-robin rotation over whichever proxies are currently healthy.
Automatic benching. After
max_failuresfailures in a row, a proxy sits out forcooldownseconds, then rejoins the pool.Safe logging. The
labelproperty prints only host and port, never your username and password.
This class isn't thread-safe. If you share one rotator across threads, wrap
get(),report_success()andreport_failure()in athreading.Lock.
Fetching with Automatic Failover
Now pair the rotator with a request function that switches proxies when one fails:
import random
import time
import requests
HEADERS = {"User-Agent": "MyLearningScraper/1.0 (educational project)"}
PROXY_ERROR_STATUS = {407, 502, 504} # what a proxy typically returns when it fails
SITE_BUSY_STATUS = {429, 500, 503} # the site itself is asking you to back off
def fetch(url, rotator, max_attempts=5, timeout=10):
"""GET a URL, switching to a different proxy whenever one fails."""
last_error = None
for attempt in range(1, max_attempts + 1):
proxy = rotator.get()
try:
response = requests.get(
url,
headers=HEADERS,
proxies={"http": proxy.url, "https": proxy.url},
timeout=timeout,
)
except requests.RequestException as exc: # ProxyError, timeouts, SSL errors...
last_error = exc
rotator.report_failure(proxy)
print(f"Attempt {attempt}: {proxy.label} failed ({type(exc).__name__})")
time.sleep(random.uniform(0.5, 1.5))
continue
if response.status_code in SITE_BUSY_STATUS: # the site says "slow down": respect it
retry_after = response.headers.get("Retry-After", "")
delay = int(retry_after) if retry_after.isdigit() else 2 ** attempt
print(f"Attempt {attempt}: site returned {response.status_code}, waiting {delay}s")
time.sleep(min(delay, 60))
continue
if response.status_code in PROXY_ERROR_STATUS: # bad proxy login, or proxy can't reach the site
rotator.report_failure(proxy)
print(f"Attempt {attempt}: {proxy.label} returned {response.status_code}")
time.sleep(random.uniform(0.5, 1.5))
continue
rotator.report_success(proxy)
response.raise_for_status() # 403, 404...: don't rotate, just raise
return response
raise RuntimeError(f"All {max_attempts} attempts failed") from last_errorThe key decisions here:
Network-level errors (connection refused, timeouts, TLS failures) count against the proxy and trigger an immediate switch.
407, 502 and 504 responses are what proxies typically return when your login is wrong or they can't reach the site, so they count against the proxy.
429, 500 and 503 come from the site, which is asking you to slow down or is having trouble. The code waits (honoring
Retry-Afterwhen provided) and doesn't blame the proxy, so a site outage can't bench your whole pool.Other 4xx errors such as 403 or 404 are raised straight away. Retrying them through a different proxy would just be hammering a site that already said no.
Plugging It into a BeautifulSoup Scraper
If you've built a scraper with BeautifulSoup, swapping in rotation is a one-line change: replace your requests.get call with fetch.
from bs4 import BeautifulSoup
PROXY_URLS = [
"http://user:[email protected]:8080",
"http://user:[email protected]:8080",
"http://user:[email protected]:8080",
]
rotator = ProxyRotator(PROXY_URLS)
response = fetch("https://books.toscrape.com/", rotator)
soup = BeautifulSoup(response.content, "html.parser")
print(soup.title.get_text(strip=True))Because the rotator keeps its state between calls, you can loop over hundreds of pages and the load will spread evenly across your healthy proxies.
Filtering Out Dead Proxies Before You Start
Proxy lists, especially free ones, are full of dead entries. It's cheaper to weed them out up front than to burn retries on them mid-crawl.
from concurrent.futures import ThreadPoolExecutor
import requests
def load_proxies(path):
"""Read one proxy URL per line, ignoring blank lines and # comments."""
with open(path, encoding="utf-8") as f:
lines = (line.strip() for line in f)
return [line for line in lines if line and not line.startswith("#")]
def check_proxy(proxy_url, timeout=8):
"""Return the exit IP if the proxy works, otherwise None."""
try:
response = requests.get(
"https://httpbin.org/ip",
proxies={"http": proxy_url, "https": proxy_url},
timeout=timeout,
)
response.raise_for_status()
return response.json()["origin"]
except (requests.RequestException, ValueError, KeyError):
return None
def healthy_proxies(candidates, workers=10):
"""Test proxies in parallel and keep only the ones that respond."""
with ThreadPoolExecutor(max_workers=workers) as pool:
exit_ips = list(pool.map(check_proxy, candidates))
return [url for url, ip in zip(candidates, exit_ips) if ip]Use it like this:
candidates = load_proxies("proxies.txt")
rotator = ProxyRotator(healthy_proxies(candidates))Rotating Proxies Asynchronously
When you need many requests in flight at once, aiohttp accepts a proxy argument on each request, which makes rotation straightforward:
pip install aiohttpimport asyncio
import itertools
import aiohttp
PROXIES = [
"http://user:[email protected]:8080",
"http://user:[email protected]:8080",
"http://user:[email protected]:8080",
]
proxy_cycle = itertools.cycle(PROXIES)
async def fetch(session, url, proxy, semaphore):
async with semaphore: # cap how many requests run at once
try:
timeout = aiohttp.ClientTimeout(total=10)
async with session.get(url, proxy=proxy, timeout=timeout) as resp:
resp.raise_for_status()
return url, resp.status, await resp.text()
except (aiohttp.ClientError, asyncio.TimeoutError) as exc:
return url, None, type(exc).__name__
async def main(urls):
semaphore = asyncio.Semaphore(5)
async with aiohttp.ClientSession() as session:
tasks = [fetch(session, url, next(proxy_cycle), semaphore) for url in urls]
return await asyncio.gather(*tasks)
if __name__ == "__main__":
results = asyncio.run(main(["https://httpbin.org/ip"] * 6))
for url, status, body in results:
print(status, body[:60])Concurrency multiplies your request rate, so keep the semaphore small and stay within what the target site allows.
A quick note on httpx
If you prefer httpx, create a client with the proxy argument:
import httpx
with httpx.Client(proxy=random_proxy(), timeout=10) as client:
print(client.get("https://httpbin.org/ip").json())Older tutorials use a proxies= argument, which was removed in httpx 0.28. Use proxy= on current versions. For rotation, build a new client per proxy.
Best Practices
Add delays and jitter. Pause between requests, ideally with a random component, so you don't hit a site in lockstep.
Rotate for reliability, not to dodge limits. Honor
429andRetry-After, and stop if you keep getting403.Always set timeouts. A proxy that hangs is worse than one that fails.
Keep sessions consistent. If a site uses cookies, create one
requests.Sessionper proxy so each exit IP carries a coherent set of cookies.Monitor your pool. Log successes and failures per proxy (by
label, never with credentials) and replace the chronic offenders.Protect credentials. Keep them out of source control and logs.
Prefer HTTPS targets. TLS keeps the content of your traffic private from the proxy operator.
Troubleshooting
Symptom | Likely cause | What to try |
|---|---|---|
| Proxy is down, or wrong host or port | Test it with |
| Wrong credentials, or special characters not URL-encoded | Re-check them and use |
| Proxy is slow or overloaded | Lower the timeout and let the rotator bench it |
| Proxy intercepts TLS, or the wrong proxy scheme is used | Use |
Frequent | You're sending requests too fast | Increase delays and respect |
Persistent | The site is refusing your traffic | Review its terms; look for an API or ask permission |
Wrapping Up
Proxy rotation in Python comes down to three ideas: describe each proxy with a URL, pick the next one with a sensible strategy (random, round robin, or a provider gateway), and handle failures so a single bad proxy never sinks your run. The ProxyRotator and fetch function above give you a solid foundation: round-robin rotation, automatic failover, benching with a cooldown, and respect for rate limits.
From here, you could extend the rotator with weighted selection (favoring faster proxies), per-proxy statistics, or a lock for multithreaded use.
Happy scraping, and please do it politely!
Found this helpful? Share it with a fellow developer or leave your questions in the comments.
Comments(0)
Join the discussion by logging into your account.
No comments yet. Be the first to comment!