Tutorial

How to Rotate Proxies in Python: A Practical Guide

Learn how to rotate proxies in Python with requests and aiohttp, from simple random choice to a reusable rotator that retries and benches bad proxies.

·
9 min read
·
AI-assisted

AI tooling disclosure

Generative AI materially contributed to this article.

Human review confirmed by the author.

How to Rotate Proxies in Python: A Practical Guide

Sooner or later, every serious scraping or data-collection project runs into the same problem: a single proxy (or your own IP address) isn't enough. It goes down, it gets slow, or you simply need to see a site from several locations. The answer is proxy rotation: spreading your requests across a pool of proxies instead of relying on just one.

In this guide you'll learn how to rotate proxies in Python, starting with a few lines of requests code and building up to a reusable rotator class that retries failed requests, benches broken proxies, and works with async code too.

Heads-up: all example proxy addresses below (203.0.113.x) come from a range reserved for documentation. Replace them with proxies you actually have access to.


Why Rotate Proxies?

There are plenty of legitimate reasons to use a pool instead of a single proxy:

  • Reliability. If one proxy dies, your job keeps running through the others.

  • Geo-testing. Check how a page, price, or ad renders from different countries.

  • Spreading permitted workloads. Large, authorized data collection can be distributed across many exit points instead of funneling through one.

  • Privacy. Avoid tying all your traffic to a single address.

Use Proxies Responsibly

Rotation is a reliability tool, not a license to ignore a site's rules. Keep these principles in mind:

  • Follow the site's Terms of Service and robots.txt. Rotating your IP doesn't make prohibited scraping permitted.

  • Treat "slow down" as an instruction. If a site rate-limits you (HTTP 429), the right response is to back off, not to hit it from a fresh IP. The code in this post honors Retry-After for exactly that reason.

  • Don't rotate your way past a block. A persistent 403 Forbidden means the site doesn't want your traffic. Look for an official API or ask for permission.

  • Never use proxies to access accounts, data, or systems you aren't authorized to use.

Proxy Basics

Types of proxies

Type

Strengths

Trade-offs

Datacenter

Fast and inexpensive

Easy for sites to identify as non-residential

Residential

IPs belong to real home connections

Slower and pricier

Mobile

Carrier-grade IPs

Usually the most expensive

Free public lists

No cost

Unreliable, slow, and potentially unsafe (see below)

A word of caution on free proxies: whoever runs a proxy can see (and potentially tamper with) any unencrypted traffic passing through it, and can log metadata even for HTTPS. Never send passwords, API keys, or personal data through a proxy you don't trust.

Proxy URL format

Python libraries describe a proxy with a single URL:

scheme://username:password@host:port

The scheme is usually http (works for both HTTP and HTTPS sites via tunneling) or socks5. Authentication is optional.

Using a Single Proxy with requests

Install the library first:

pip install requests

Then pass a proxies dictionary to any request:

import requests

proxy_url = "http://user:[email protected]:8080"
proxies = {"http": proxy_url, "https": proxy_url}

response = requests.get("https://httpbin.org/ip", proxies=proxies, timeout=10)
print(response.json())      # {'origin': '203.0.113.10'}

The httpbin.org/ip endpoint echoes the IP address it sees, which makes it handy for confirming that your traffic really goes through the proxy.

A few details worth knowing:

  • Both dictionary keys usually point to the same http:// proxy URL. The key says which target scheme the proxy is used for; the proxy URL's own scheme describes how you talk to the proxy.

  • requests also reads the HTTP_PROXY and HTTPS_PROXY environment variables, but a proxies= argument you pass explicitly takes precedence.

  • Always set a timeout. Dead proxies often hang rather than fail fast.

Special characters in credentials

If your username or password contains characters like @, : or /, URL-encode them. Otherwise the proxy URL is parsed incorrectly and requests typically fails with an InvalidURL error:

from urllib.parse import quote

username = quote("[email protected]", safe="")
password = quote("p@ss:word/1", safe="")
proxy_url = f"http://{username}:{password}@203.0.113.10:8080"

print(proxy_url)

Better still, don't hardcode credentials at all. Load them from environment variables or a secrets manager.

SOCKS proxies

For SOCKS support, install the extra and change the scheme:

pip install "requests[socks]"
proxy_url = "socks5h://user:[email protected]:1080"

The socks5h variant resolves DNS through the proxy as well, which avoids leaking your lookups to your own resolver.

Rotation Strategies

1. Random choice

The simplest approach: pick any proxy for each request.

import itertools
import random

PROXIES = [
    "http://user:[email protected]:8080",
    "http://user:[email protected]:8080",
    "http://user:[email protected]:8080",
]


def random_proxy():
    return random.choice(PROXIES)

It's easy, but you can't guarantee an even spread, and the same proxy may be chosen several times in a row.

2. Round robin

itertools.cycle walks through the list in order and loops forever, so every proxy gets an equal share:

proxy_cycle = itertools.cycle(PROXIES)


def next_proxy():
    return next(proxy_cycle)

3. Let the provider rotate for you

Many paid services expose a single "gateway" endpoint that assigns a different exit IP behind the scenes. In that case your code needs no rotation logic at all:

gateway = "http://user:[email protected]:8000"
proxies = {"http": gateway, "https": gateway}

Check your provider's documentation. Some rotate on every request, others keep a "sticky" IP for a few minutes, which is useful when a multi-step flow (like paging through results) should come from one address.

Which should you use?

Strategy

Best for

Weakness

Random

Quick scripts

Uneven distribution

Round robin

Fair, predictable spread

Doesn't notice dead proxies

Provider gateway

Large pools with zero maintenance

Costs money, less control

Rotator with health tracking

Anything you'll run repeatedly

A bit more code (below)

Building a Reusable Rotator

Random and round-robin both share a flaw: they keep handing out a proxy even after it has stopped working. A better rotator tracks failures and temporarily benches proxies that misbehave.

import time
from dataclasses import dataclass
from urllib.parse import urlsplit


@dataclass
class Proxy:
    url: str
    failures: int = 0
    banned_until: float = 0.0

    @property
    def label(self) -> str:
        """Host and port only, so credentials never end up in logs."""
        return urlsplit(self.url).netloc.rpartition("@")[2]


class ProxyRotator:
    def __init__(self, proxy_urls, max_failures=3, cooldown=300):
        self.proxies = [Proxy(url) for url in proxy_urls]
        self.max_failures = max_failures      # consecutive failures before benching
        self.cooldown = cooldown              # seconds a benched proxy sits out
        self._counter = 0

    def get(self) -> Proxy:
        """Return the next healthy proxy in round-robin order."""
        now = time.time()
        healthy = [p for p in self.proxies if p.banned_until <= now]
        if not healthy:
            raise RuntimeError("No healthy proxies left")
        proxy = healthy[self._counter % len(healthy)]
        self._counter += 1
        return proxy

    def report_success(self, proxy: Proxy) -> None:
        proxy.failures = 0

    def report_failure(self, proxy: Proxy) -> None:
        proxy.failures += 1
        if proxy.failures >= self.max_failures:
            proxy.banned_until = time.time() + self.cooldown
            proxy.failures = 0

What this gives you:

  • Round-robin rotation over whichever proxies are currently healthy.

  • Automatic benching. After max_failures failures in a row, a proxy sits out for cooldown seconds, then rejoins the pool.

  • Safe logging. The label property prints only host and port, never your username and password.

This class isn't thread-safe. If you share one rotator across threads, wrap get(), report_success() and report_failure() in a threading.Lock.

Fetching with Automatic Failover

Now pair the rotator with a request function that switches proxies when one fails:

import random
import time

import requests

HEADERS = {"User-Agent": "MyLearningScraper/1.0 (educational project)"}
PROXY_ERROR_STATUS = {407, 502, 504}     # what a proxy typically returns when it fails
SITE_BUSY_STATUS = {429, 500, 503}       # the site itself is asking you to back off


def fetch(url, rotator, max_attempts=5, timeout=10):
    """GET a URL, switching to a different proxy whenever one fails."""
    last_error = None

    for attempt in range(1, max_attempts + 1):
        proxy = rotator.get()
        try:
            response = requests.get(
                url,
                headers=HEADERS,
                proxies={"http": proxy.url, "https": proxy.url},
                timeout=timeout,
            )
        except requests.RequestException as exc:   # ProxyError, timeouts, SSL errors...
            last_error = exc
            rotator.report_failure(proxy)
            print(f"Attempt {attempt}: {proxy.label} failed ({type(exc).__name__})")
            time.sleep(random.uniform(0.5, 1.5))
            continue

        if response.status_code in SITE_BUSY_STATUS:     # the site says "slow down": respect it
            retry_after = response.headers.get("Retry-After", "")
            delay = int(retry_after) if retry_after.isdigit() else 2 ** attempt
            print(f"Attempt {attempt}: site returned {response.status_code}, waiting {delay}s")
            time.sleep(min(delay, 60))
            continue

        if response.status_code in PROXY_ERROR_STATUS:   # bad proxy login, or proxy can't reach the site
            rotator.report_failure(proxy)
            print(f"Attempt {attempt}: {proxy.label} returned {response.status_code}")
            time.sleep(random.uniform(0.5, 1.5))
            continue

        rotator.report_success(proxy)
        response.raise_for_status()                # 403, 404...: don't rotate, just raise
        return response

    raise RuntimeError(f"All {max_attempts} attempts failed") from last_error

The key decisions here:

  • Network-level errors (connection refused, timeouts, TLS failures) count against the proxy and trigger an immediate switch.

  • 407, 502 and 504 responses are what proxies typically return when your login is wrong or they can't reach the site, so they count against the proxy.

  • 429, 500 and 503 come from the site, which is asking you to slow down or is having trouble. The code waits (honoring Retry-After when provided) and doesn't blame the proxy, so a site outage can't bench your whole pool.

  • Other 4xx errors such as 403 or 404 are raised straight away. Retrying them through a different proxy would just be hammering a site that already said no.

Plugging It into a BeautifulSoup Scraper

If you've built a scraper with BeautifulSoup, swapping in rotation is a one-line change: replace your requests.get call with fetch.

from bs4 import BeautifulSoup

PROXY_URLS = [
    "http://user:[email protected]:8080",
    "http://user:[email protected]:8080",
    "http://user:[email protected]:8080",
]

rotator = ProxyRotator(PROXY_URLS)

response = fetch("https://books.toscrape.com/", rotator)
soup = BeautifulSoup(response.content, "html.parser")
print(soup.title.get_text(strip=True))

Because the rotator keeps its state between calls, you can loop over hundreds of pages and the load will spread evenly across your healthy proxies.

Filtering Out Dead Proxies Before You Start

Proxy lists, especially free ones, are full of dead entries. It's cheaper to weed them out up front than to burn retries on them mid-crawl.

from concurrent.futures import ThreadPoolExecutor

import requests


def load_proxies(path):
    """Read one proxy URL per line, ignoring blank lines and # comments."""
    with open(path, encoding="utf-8") as f:
        lines = (line.strip() for line in f)
        return [line for line in lines if line and not line.startswith("#")]


def check_proxy(proxy_url, timeout=8):
    """Return the exit IP if the proxy works, otherwise None."""
    try:
        response = requests.get(
            "https://httpbin.org/ip",
            proxies={"http": proxy_url, "https": proxy_url},
            timeout=timeout,
        )
        response.raise_for_status()
        return response.json()["origin"]
    except (requests.RequestException, ValueError, KeyError):
        return None


def healthy_proxies(candidates, workers=10):
    """Test proxies in parallel and keep only the ones that respond."""
    with ThreadPoolExecutor(max_workers=workers) as pool:
        exit_ips = list(pool.map(check_proxy, candidates))
    return [url for url, ip in zip(candidates, exit_ips) if ip]

Use it like this:

candidates = load_proxies("proxies.txt")
rotator = ProxyRotator(healthy_proxies(candidates))

Rotating Proxies Asynchronously

When you need many requests in flight at once, aiohttp accepts a proxy argument on each request, which makes rotation straightforward:

pip install aiohttp
import asyncio
import itertools

import aiohttp

PROXIES = [
    "http://user:[email protected]:8080",
    "http://user:[email protected]:8080",
    "http://user:[email protected]:8080",
]
proxy_cycle = itertools.cycle(PROXIES)


async def fetch(session, url, proxy, semaphore):
    async with semaphore:                          # cap how many requests run at once
        try:
            timeout = aiohttp.ClientTimeout(total=10)
            async with session.get(url, proxy=proxy, timeout=timeout) as resp:
                resp.raise_for_status()
                return url, resp.status, await resp.text()
        except (aiohttp.ClientError, asyncio.TimeoutError) as exc:
            return url, None, type(exc).__name__


async def main(urls):
    semaphore = asyncio.Semaphore(5)
    async with aiohttp.ClientSession() as session:
        tasks = [fetch(session, url, next(proxy_cycle), semaphore) for url in urls]
        return await asyncio.gather(*tasks)


if __name__ == "__main__":
    results = asyncio.run(main(["https://httpbin.org/ip"] * 6))
    for url, status, body in results:
        print(status, body[:60])

Concurrency multiplies your request rate, so keep the semaphore small and stay within what the target site allows.

A quick note on httpx

If you prefer httpx, create a client with the proxy argument:

import httpx

with httpx.Client(proxy=random_proxy(), timeout=10) as client:
    print(client.get("https://httpbin.org/ip").json())

Older tutorials use a proxies= argument, which was removed in httpx 0.28. Use proxy= on current versions. For rotation, build a new client per proxy.

Best Practices

  • Add delays and jitter. Pause between requests, ideally with a random component, so you don't hit a site in lockstep.

  • Rotate for reliability, not to dodge limits. Honor 429 and Retry-After, and stop if you keep getting 403.

  • Always set timeouts. A proxy that hangs is worse than one that fails.

  • Keep sessions consistent. If a site uses cookies, create one requests.Session per proxy so each exit IP carries a coherent set of cookies.

  • Monitor your pool. Log successes and failures per proxy (by label, never with credentials) and replace the chronic offenders.

  • Protect credentials. Keep them out of source control and logs.

  • Prefer HTTPS targets. TLS keeps the content of your traffic private from the proxy operator.

Troubleshooting

Symptom

Likely cause

What to try

ProxyError / connection refused

Proxy is down, or wrong host or port

Test it with check_proxy; verify the address

407 Proxy Authentication Required

Wrong credentials, or special characters not URL-encoded

Re-check them and use urllib.parse.quote

ConnectTimeout / ReadTimeout

Proxy is slow or overloaded

Lower the timeout and let the rotator bench it

SSLError

Proxy intercepts TLS, or the wrong proxy scheme is used

Use http:// for an HTTP proxy URL; avoid proxies that tamper with certificates

Frequent 429

You're sending requests too fast

Increase delays and respect Retry-After

Persistent 403

The site is refusing your traffic

Review its terms; look for an API or ask permission

Wrapping Up

Proxy rotation in Python comes down to three ideas: describe each proxy with a URL, pick the next one with a sensible strategy (random, round robin, or a provider gateway), and handle failures so a single bad proxy never sinks your run. The ProxyRotator and fetch function above give you a solid foundation: round-robin rotation, automatic failover, benching with a cooldown, and respect for rate limits.

From here, you could extend the rotator with weighted selection (favoring faster proxies), per-proxy statistics, or a lock for multithreaded use.

Happy scraping, and please do it politely!


Found this helpful? Share it with a fellow developer or leave your questions in the comments.

Comments(0)

Join the discussion by logging into your account.

No comments yet. Be the first to comment!

Bhavya Arora

Passionate developer sharing knowledge about modern web technologies and best practices.

Subscribe to Bhavya Arora's Newsletter

Direct email dispatches when new stories are published. Zero algorithms.