--- title: "Rotating User Agents and Headers in Python: A Practical Guide" author: "Bhavya Arora" published: "2026-11-17T15:57:39.891Z" updated: "2026-11-17T15:57:39.891Z" url: "https://zyvop.com/rotating-user-agents-and-headers-in-python-a-practical-guide-7p6sc" canonical: "https://zyvop.com/rotating-user-agents-and-headers-in-python-a-practical-guide-7p6sc" tags: ["Web Scraping","Python","http-headers","Requests","user-agent"] --- # Rotating User Agents and Headers in Python: A Practical Guide Learn how to rotate User-Agent strings and HTTP headers in Python using requests, httpx, and Scrapy with consistent browser profiles. ![Cover image]() Every request your scraper sends carries a small introduction: _who am I, and what can I handle?_ Left alone, a Python script introduces itself as `python-requests/2.34.2`, and some sites reject that on sight, even when they would happily serve a polite scraper. The usual advice is to "rotate your user agent", and the usual code picks a random string from a list. That is a start, but on its own it is often _worse_ than doing nothing. A Chrome user agent that arrives without the other headers Chrome sends is a client that doesn't exist in the real world. This guide shows a better approach: rotate **complete, consistent header profiles**, **per session** rather than per request, and **check what you actually send**. > **Version note:** tested with Python 3.13, `requests` 2.34, `httpx` 0.28, Scrapy 2.20, `curl_cffi` 0.17 and `fake-useragent` 2.2. The browser versions in the code (Chrome 152, Firefox 155) are a few releases behind the current ones (Chrome 157, with 158 rolling out, and Firefox 160 in mid-November 2026). Browsers update every few weeks, so treat the versions as placeholders to refresh. * * * ## What the Server Sees You can't improve what you can't see, so start by building a tiny server that echoes back the headers it receives. Save this as `echo_server.py`: ```python import json from http.server import BaseHTTPRequestHandler, HTTPServer class Echo(BaseHTTPRequestHandler): def do_GET(self): body = json.dumps(dict(self.headers.items()), indent=2).encode() self.send_response(200) self.send_header("Content-Type", "application/json") self.send_header("Content-Length", str(len(body))) self.end_headers() self.wfile.write(body) def log_message(self, *args): # keep the terminal quiet pass HTTPServer(("127.0.0.1", 8000), Echo).serve_forever() ``` Run it in one terminal with `python echo_server.py`, install the libraries in another (`pip install requests httpx`), and look at what a plain `requests` call sends: ```python import requests print(requests.get("http://127.0.0.1:8000", timeout=10).text) ``` ```json { "Host": "127.0.0.1:8000", "User-Agent": "python-requests/2.34.2", "Accept-Encoding": "gzip, deflate, br, zstd", "Accept": "*/*", "Connection": "keep-alive" } ``` (Your `Accept-Encoding` depends on which compression libraries are installed. That detail matters later.) Compare this with what Chrome sends for an ordinary page visit: a dozen headers, including language preferences, a detailed `Accept` list and several `sec-` headers. A server that looks at all of them sees an obvious difference. ## What a Browser Sends You don't need to memorize every header, but you should know the ones that matter: | Header | What it says | Notes | | --- | --- | --- | | `User-Agent` | Browser, major version and OS family | Chrome freezes the minor version (`152.0.0.0`) and Safari freezes the macOS version (`10_15_7`), so little else varies | | `Accept` | Content types the client can handle | Browsers send a long, browser-specific list; libraries send `*/*` | | `Accept-Language` | Preferred languages | Libraries often send none at all | | `Accept-Encoding` | Compression the client can decode | Must match what your library can actually decompress | | `sec-ch-ua`, `sec-ch-ua-mobile`, `sec-ch-ua-platform` | Client Hints repeating the browser, device type and OS | Sent by Chrome and other Chromium browsers, **not** by Firefox or Safari | | `Sec-Fetch-Site`, `-Mode`, `-Dest`, `-User` | How the request was triggered | `none` for a typed URL, `same-origin` for a click within a site | | `Referer` | The page you came from | Absent on a first visit | The important idea is **consistency**. The user agent, the Client Hints, the platform and the browser version must all tell the same story. ## Level 1: Rotate the User-Agent The simplest version picks a user agent at random for each request: ```python import random import requests USER_AGENTS = [ "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/152.0.0.0 Safari/537.36", "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/152.0.0.0 Safari/537.36", "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:155.0) Gecko/20100101 Firefox/155.0", ] def get(url): headers = {"User-Agent": random.choice(USER_AGENTS)} return requests.get(url, headers=headers, timeout=10) for _ in range(3): print(get("http://127.0.0.1:8000").json()["User-Agent"]) ``` This works as far as it goes, and a lenient site may need nothing more. But look at what else arrives: `Accept: */*`, no `Accept-Language`, no `sec-ch-ua`. Every one of those requests claims to be a modern browser while sending the headers of a Python library. ## Level 2: Rotate Whole Header Profiles A **profile** is a complete header set copied from a real browser. To capture one, open your browser's developer tools, go to the **Network** tab, load a page, right-click the first request and choose **Copy as cURL**. Keep headers like the ones in the profile below, and drop `Cookie`, `Host` and `Accept-Encoding`, which your library should manage itself. Create `profiles.py`: ```python import re CHROME_WINDOWS = { "sec-ch-ua": '"Chromium";v="152", "Not?A_Brand";v="24", "Google Chrome";v="152"', "sec-ch-ua-mobile": "?0", "sec-ch-ua-platform": '"Windows"', "Upgrade-Insecure-Requests": "1", "User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/152.0.0.0 Safari/537.36", "Accept": "text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.7", "Sec-Fetch-Site": "none", "Sec-Fetch-Mode": "navigate", "Sec-Fetch-User": "?1", "Sec-Fetch-Dest": "document", "Accept-Language": "en-US,en;q=0.9", } CHROME_MACOS = { **CHROME_WINDOWS, "sec-ch-ua-platform": '"macOS"', "User-Agent": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/152.0.0.0 Safari/537.36", } PROFILES = [CHROME_WINDOWS, CHROME_MACOS] ``` These two are the headers that the Chrome 152 profile of `curl_cffi` (more on that library below) sent to my echo server, so use them as a starting point and replace them with a capture from your own browser when you can. Add Firefox or Safari profiles the same way. They have no `sec-ch-ua` headers, which is exactly why you copy them whole instead of building them by hand. Hand-built or mixed-up profiles are easy to get wrong, so add a quick linter to the same file. It catches the mismatches that give a fake identity away: ```python def check_profile(headers): """Return a list of ways this header set contradicts itself.""" h = {name.lower(): value for name, value in headers.items()} ua = h.get("user-agent", "") problems = [] if "Chrome/" not in ua: if "sec-ch-ua" in h: problems.append("sec-ch-ua headers without a Chrome user agent") return problems if "sec-ch-ua" not in h: return ["Chrome user agent but no sec-ch-ua headers"] major = re.search(r"Chrome/(\d+)", ua).group(1) if f'v="{major}"' not in h["sec-ch-ua"]: problems.append(f"user agent says Chrome {major} but sec-ch-ua disagrees") platform = h.get("sec-ch-ua-platform", "").strip('"') expected = "Windows" if "Windows" in ua else "macOS" if "Macintosh" in ua else "Linux" if platform != expected: problems.append(f"user agent says {expected} but sec-ch-ua-platform says {platform or 'nothing'}") return problems for profile in PROFILES: assert check_profile(profile) == [] ``` ## Level 3: Rotate Sessions, Not Requests A real visitor doesn't change browsers between page views. If you rotate on _every_ request, one cookie-carrying session appears to switch from Windows Chrome to a Mac a second later. Rotate per **session** instead: choose a profile, keep it for a few dozen requests, then start over with a fresh identity and fresh cookies. Save this as `browser_session.py` next to `profiles.py`: ```python import random from urllib.parse import urlsplit import requests from profiles import PROFILES class BrowserSession: """A requests session that behaves like one consistent browser.""" def __init__(self, profiles=PROFILES, rotate_after=(20, 40)): self.profiles = profiles self.rotate_after = rotate_after self._new_identity() def _new_identity(self): self.profile = random.choice(self.profiles) self.session = requests.Session() # a new session means new cookies self.session.headers.update(self.profile) # replaces User-Agent and Accept self.limit = random.randint(*self.rotate_after) self.count = 0 self.last_url = None def get(self, url, **kwargs): if self.count >= self.limit: self._new_identity() headers = {} if self.last_url: # behave like a click from the last page same_site = urlsplit(url).netloc == urlsplit(self.last_url).netloc headers["Referer"] = self.last_url headers["Sec-Fetch-Site"] = "same-origin" if same_site else "cross-site" kwargs.setdefault("timeout", 10) response = self.session.get(url, headers=headers, **kwargs) self.count += 1 self.last_url = url return response if __name__ == "__main__": browser = BrowserSession() for page in range(1, 4): reply = browser.get(f"http://127.0.0.1:8000/page-{page}").json() print(reply["User-Agent"][:60], "| Referer:", reply.get("Referer")) ``` Three details are worth noticing: - `session.headers.update(profile)` overwrites the library's `User-Agent` and `Accept` and adds everything else. It leaves `Accept-Encoding` to `requests` for a reason covered below. - **The first request has no** `Referer` and `Sec-Fetch-Site: none`, like a typed address. Later requests look like clicks. - **A new** `Session` **is a new identity.** Cookies don't leak from one "visitor" to the next. If you also use proxies (see our guide to rotating proxies in Python), give each identity its own proxy and keep the pair together. Five different browsers behind one IP address looks just as odd as one browser jumping between five IP addresses mid-session. ## The Same Idea in httpx The profile dictionaries work unchanged with `httpx`: ```python import random import httpx from profiles import PROFILES profile = random.choice(PROFILES) with httpx.Client(headers=profile, timeout=10) as client: first = client.get("http://127.0.0.1:8000/page-1") print(first.json()["User-Agent"]) ``` `httpx` keeps its own defaults (`Connection`, `Accept-Encoding`) and replaces `User-Agent` and `Accept` with yours. Like `requests`, it sends headers in its own order rather than a browser's. ## Rotating Profiles in Scrapy In Scrapy, a downloader middleware is the right place. Put `profiles.py` inside your project package (here it's called `myproject`), then create the middleware in `myproject/middlewares.py`: ```python import random from myproject.profiles import PROFILES class RotateProfileMiddleware: """Give every request a coherent browser header set.""" def process_request(self, request): profile = request.meta.setdefault("profile", random.choice(PROFILES)) for name, value in profile.items(): request.headers[name] = value ``` Enable it in `settings.py`: ```python DOWNLOADER_MIDDLEWARES = { "myproject.middlewares.RotateProfileMiddleware": 543, } ``` The number matters: 543 runs _after_ Scrapy's built-in header middlewares (at 400 and 500), so your values replace their defaults. This also overrides the `USER_AGENT` setting. Because the chosen profile is stored in `request.meta["profile"]`, a retried or redirected request keeps the same identity. New requests created with `response.follow()` don't inherit `meta`, so to pin a whole chain of requests to one identity, pass it along explicitly: ```python yield response.follow(url, callback=self.parse_page, meta={"profile": response.meta["profile"]}) ``` Scrapy normalizes header names (`sec-ch-ua` goes out as `Sec-Ch-Ua`), one more small difference from a real browser that headers alone can't fix. ## What About fake-useragent? The [fake-useragent]() package gives you a ready-made list of user agents, and it's handy for quick experiments: ```python from fake_useragent import UserAgent ua = UserAgent(browsers=["Chrome", "Firefox"], os=["Windows", "Mac OS X", "Linux"], platforms=["desktop"]) print(ua.random) ``` Treat it with care. When I tested version 2.2.0: - **The default is mostly mobile.** Mobile devices make up about 69% of its weighted data; in 3,000 draws from plain `UserAgent().random`, 2,070 were mobile, including in-app browsers such as the Google and Facebook apps. Always pass `platforms=["desktop"]` unless you want mobile pages. - **The data is old.** Its newest Chrome is 136 and its newest Firefox 138, both from spring 2025, while real browsers were at 157 and 160. It also contains Chrome versions as old as 65. - **Failures are easy to miss.** Asking for a `min_version` it has no data for doesn't raise an exception. It logs a short error and returns a fallback Edge 122 user agent. And a user agent alone is only one line of a profile. Drop one of its strings into a profile and the linter shows the problem: ```python from fake_useragent import UserAgent from profiles import CHROME_WINDOWS, check_profile ua = UserAgent(browsers=["Chrome"], os=["Windows"], platforms=["desktop"]) mixed = {**CHROME_WINDOWS, "User-Agent": ua.random} print(check_profile(mixed)) # ['user agent says Chrome 135 but sec-ch-ua disagrees'] (the version varies) ``` Your own captured profiles, refreshed every few months, beat a random list. ## The Accept-Encoding Trap This is the one header you shouldn't copy from your browser. Chrome advertises `gzip, deflate, br, zstd`. If you paste that into a `requests` setup that doesn't have the `brotli` package installed, the server may answer in Brotli and `response.text` comes back as binary garbage. I reproduced exactly this. Leave `Accept-Encoding` to the library, as the code above does, or install `brotli` and `zstandard` first. ## Beyond Headers Headers are only one signal. Sites can also look at the TLS handshake your client performs, HTTP/2 details, the order and capitalization of headers, your IP reputation and whether you run their JavaScript. Python's `requests` can't change most of that, and no header profile can hide it. Some libraries address the lower layers. `curl_cffi`, for example, impersonates a browser's connection and sends a matching, complete header set: ```python from curl_cffi import requests as cffi_requests response = cffi_requests.get("http://127.0.0.1:8000", impersonate="chrome") print(response.json()["user-agent"]) ``` If a site has gone to the trouble of blocking automated traffic, more disguise only starts an arms race. ## Using This Responsibly Header rotation is a tool for being _compatible_, not for being _invisible_. - **Prefer honesty where you can.** A descriptive agent such as `MyBot/1.0 (+https://example.com/bot; me@example.com)` lets site owners contact you instead of blocking you. Use it unless a site rejects it for no good reason. - **Never rotate to dodge a rule.** `robots.txt` rules are addressed to crawlers by name, and a `*` group applies to everyone. Changing your identity to slip past a `Disallow`, a rate limit or a ban you were given is circumvention, and it can turn a terms-of-service problem into a bigger one. Our guide "Is Web Scraping Legal?" explains why. - **Don't raise your volume.** Rotation doesn't change how many requests you send. Keep your delays and respect `429` responses. - **Legitimate uses exist.** Making a permitted scraper work with a site that rejects library agents, or testing how your own site responds to different browsers. ## Troubleshooting | Symptom | Likely cause | What to try | | --- | --- | --- | | `403` even with a full browser profile | The block uses your IP, TLS fingerprint or a JavaScript check, not your headers | Slow down and re-read the site's rules before trying anything more | | `response.text` is binary garbage | You advertised `br` or `zstd` without the matching package | Drop `Accept-Encoding` or install `brotli` and `zstandard` | | Parser breaks on some requests | You're getting mobile and desktop HTML at random | Use desktop-only profiles | | CAPTCHAs appear after rotating | Identity changes mid-session, or one IP shows many browsers | Rotate per session and pair each identity with one proxy | | `check_profile` reports a mismatch | Header set was mixed from different sources | Recapture the whole set from one browser | | Scrapy ignores your `USER_AGENT` setting | The middleware overwrites it | That's intended; edit the profiles instead | ## Where to Go from Here You now have a pattern you can trust: capture real header sets, lint them, rotate them per session, and check what actually leaves your machine with the echo server. From here: - Combine profiles with **rotating proxies** and a polite delay (see our guides on both). - If a site needs JavaScript, use a real browser instead of imitating one (see our Playwright guide). Happy scraping, and please do it politely! * * * _Found this helpful? Share it with a fellow developer or leave your questions in the comments._