
Every request your scraper sends carries a small introduction: who am I, and what can I handle? Left alone, a Python script introduces itself as python-requests/2.34.2, and some sites reject that on sight, even when they would happily serve a polite scraper.
The usual advice is to "rotate your user agent", and the usual code picks a random string from a list. That is a start, but on its own it is often worse than doing nothing. A Chrome user agent that arrives without the other headers Chrome sends is a client that doesn't exist in the real world. This guide shows a better approach: rotate complete, consistent header profiles, per session rather than per request, and check what you actually send.
Version note: tested with Python 3.13,
requests2.34,httpx0.28, Scrapy 2.20,curl_cffi0.17 andfake-useragent2.2. The browser versions in the code (Chrome 152, Firefox 155) are a few releases behind the current ones (Chrome 157, with 158 rolling out, and Firefox 160 in mid-November 2026). Browsers update every few weeks, so treat the versions as placeholders to refresh.
What the Server Sees
You can't improve what you can't see, so start by building a tiny server that echoes back the headers it receives. Save this as echo_server.py:
import json
from http.server import BaseHTTPRequestHandler, HTTPServer
class Echo(BaseHTTPRequestHandler):
def do_GET(self):
body = json.dumps(dict(self.headers.items()), indent=2).encode()
self.send_response(200)
self.send_header("Content-Type", "application/json")
self.send_header("Content-Length", str(len(body)))
self.end_headers()
self.wfile.write(body)
def log_message(self, *args): # keep the terminal quiet
pass
HTTPServer(("127.0.0.1", 8000), Echo).serve_forever()Run it in one terminal with python echo_server.py, install the libraries in another (pip install requests httpx), and look at what a plain requests call sends:
import requests
print(requests.get("http://127.0.0.1:8000", timeout=10).text){
"Host": "127.0.0.1:8000",
"User-Agent": "python-requests/2.34.2",
"Accept-Encoding": "gzip, deflate, br, zstd",
"Accept": "*/*",
"Connection": "keep-alive"
}(Your Accept-Encoding depends on which compression libraries are installed. That detail matters later.) Compare this with what Chrome sends for an ordinary page visit: a dozen headers, including language preferences, a detailed Accept list and several sec- headers. A server that looks at all of them sees an obvious difference.
What a Browser Sends
You don't need to memorize every header, but you should know the ones that matter:
Header | What it says | Notes |
|---|---|---|
| Browser, major version and OS family | Chrome freezes the minor version ( |
| Content types the client can handle | Browsers send a long, browser-specific list; libraries send |
| Preferred languages | Libraries often send none at all |
| Compression the client can decode | Must match what your library can actually decompress |
| Client Hints repeating the browser, device type and OS | Sent by Chrome and other Chromium browsers, not by Firefox or Safari |
| How the request was triggered |
|
| The page you came from | Absent on a first visit |
The important idea is consistency. The user agent, the Client Hints, the platform and the browser version must all tell the same story.
Level 1: Rotate the User-Agent
The simplest version picks a user agent at random for each request:
import random
import requests
USER_AGENTS = [
"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/152.0.0.0 Safari/537.36",
"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/152.0.0.0 Safari/537.36",
"Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:155.0) Gecko/20100101 Firefox/155.0",
]
def get(url):
headers = {"User-Agent": random.choice(USER_AGENTS)}
return requests.get(url, headers=headers, timeout=10)
for _ in range(3):
print(get("http://127.0.0.1:8000").json()["User-Agent"])This works as far as it goes, and a lenient site may need nothing more. But look at what else arrives: Accept: */*, no Accept-Language, no sec-ch-ua. Every one of those requests claims to be a modern browser while sending the headers of a Python library.
Level 2: Rotate Whole Header Profiles
A profile is a complete header set copied from a real browser. To capture one, open your browser's developer tools, go to the Network tab, load a page, right-click the first request and choose Copy as cURL. Keep headers like the ones in the profile below, and drop Cookie, Host and Accept-Encoding, which your library should manage itself.
Create profiles.py:
import re
CHROME_WINDOWS = {
"sec-ch-ua": '"Chromium";v="152", "Not?A_Brand";v="24", "Google Chrome";v="152"',
"sec-ch-ua-mobile": "?0",
"sec-ch-ua-platform": '"Windows"',
"Upgrade-Insecure-Requests": "1",
"User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/152.0.0.0 Safari/537.36",
"Accept": "text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.7",
"Sec-Fetch-Site": "none",
"Sec-Fetch-Mode": "navigate",
"Sec-Fetch-User": "?1",
"Sec-Fetch-Dest": "document",
"Accept-Language": "en-US,en;q=0.9",
}
CHROME_MACOS = {
**CHROME_WINDOWS,
"sec-ch-ua-platform": '"macOS"',
"User-Agent": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/152.0.0.0 Safari/537.36",
}
PROFILES = [CHROME_WINDOWS, CHROME_MACOS]These two are the headers that the Chrome 152 profile of curl_cffi (more on that library below) sent to my echo server, so use them as a starting point and replace them with a capture from your own browser when you can. Add Firefox or Safari profiles the same way. They have no sec-ch-ua headers, which is exactly why you copy them whole instead of building them by hand.
Hand-built or mixed-up profiles are easy to get wrong, so add a quick linter to the same file. It catches the mismatches that give a fake identity away:
def check_profile(headers):
"""Return a list of ways this header set contradicts itself."""
h = {name.lower(): value for name, value in headers.items()}
ua = h.get("user-agent", "")
problems = []
if "Chrome/" not in ua:
if "sec-ch-ua" in h:
problems.append("sec-ch-ua headers without a Chrome user agent")
return problems
if "sec-ch-ua" not in h:
return ["Chrome user agent but no sec-ch-ua headers"]
major = re.search(r"Chrome/(\d+)", ua).group(1)
if f'v="{major}"' not in h["sec-ch-ua"]:
problems.append(f"user agent says Chrome {major} but sec-ch-ua disagrees")
platform = h.get("sec-ch-ua-platform", "").strip('"')
expected = "Windows" if "Windows" in ua else "macOS" if "Macintosh" in ua else "Linux"
if platform != expected:
problems.append(f"user agent says {expected} but sec-ch-ua-platform says {platform or 'nothing'}")
return problems
for profile in PROFILES:
assert check_profile(profile) == []Level 3: Rotate Sessions, Not Requests
A real visitor doesn't change browsers between page views. If you rotate on every request, one cookie-carrying session appears to switch from Windows Chrome to a Mac a second later. Rotate per session instead: choose a profile, keep it for a few dozen requests, then start over with a fresh identity and fresh cookies. Save this as browser_session.py next to profiles.py:
import random
from urllib.parse import urlsplit
import requests
from profiles import PROFILES
class BrowserSession:
"""A requests session that behaves like one consistent browser."""
def __init__(self, profiles=PROFILES, rotate_after=(20, 40)):
self.profiles = profiles
self.rotate_after = rotate_after
self._new_identity()
def _new_identity(self):
self.profile = random.choice(self.profiles)
self.session = requests.Session() # a new session means new cookies
self.session.headers.update(self.profile) # replaces User-Agent and Accept
self.limit = random.randint(*self.rotate_after)
self.count = 0
self.last_url = None
def get(self, url, **kwargs):
if self.count >= self.limit:
self._new_identity()
headers = {}
if self.last_url: # behave like a click from the last page
same_site = urlsplit(url).netloc == urlsplit(self.last_url).netloc
headers["Referer"] = self.last_url
headers["Sec-Fetch-Site"] = "same-origin" if same_site else "cross-site"
kwargs.setdefault("timeout", 10)
response = self.session.get(url, headers=headers, **kwargs)
self.count += 1
self.last_url = url
return response
if __name__ == "__main__":
browser = BrowserSession()
for page in range(1, 4):
reply = browser.get(f"http://127.0.0.1:8000/page-{page}").json()
print(reply["User-Agent"][:60], "| Referer:", reply.get("Referer"))Three details are worth noticing:
session.headers.update(profile)overwrites the library'sUser-AgentandAcceptand adds everything else. It leavesAccept-Encodingtorequestsfor a reason covered below.The first request has no
RefererandSec-Fetch-Site: none, like a typed address. Later requests look like clicks.A new
Sessionis a new identity. Cookies don't leak from one "visitor" to the next.
If you also use proxies (see our guide to rotating proxies in Python), give each identity its own proxy and keep the pair together. Five different browsers behind one IP address looks just as odd as one browser jumping between five IP addresses mid-session.
The Same Idea in httpx
The profile dictionaries work unchanged with httpx:
import random
import httpx
from profiles import PROFILES
profile = random.choice(PROFILES)
with httpx.Client(headers=profile, timeout=10) as client:
first = client.get("http://127.0.0.1:8000/page-1")
print(first.json()["User-Agent"])httpx keeps its own defaults (Connection, Accept-Encoding) and replaces User-Agent and Accept with yours. Like requests, it sends headers in its own order rather than a browser's.
Rotating Profiles in Scrapy
In Scrapy, a downloader middleware is the right place. Put profiles.py inside your project package (here it's called myproject), then create the middleware in myproject/middlewares.py:
import random
from myproject.profiles import PROFILES
class RotateProfileMiddleware:
"""Give every request a coherent browser header set."""
def process_request(self, request):
profile = request.meta.setdefault("profile", random.choice(PROFILES))
for name, value in profile.items():
request.headers[name] = valueEnable it in settings.py:
DOWNLOADER_MIDDLEWARES = {
"myproject.middlewares.RotateProfileMiddleware": 543,
}The number matters: 543 runs after Scrapy's built-in header middlewares (at 400 and 500), so your values replace their defaults. This also overrides the USER_AGENT setting. Because the chosen profile is stored in request.meta["profile"], a retried or redirected request keeps the same identity. New requests created with response.follow() don't inherit meta, so to pin a whole chain of requests to one identity, pass it along explicitly:
yield response.follow(url, callback=self.parse_page, meta={"profile": response.meta["profile"]})Scrapy normalizes header names (sec-ch-ua goes out as Sec-Ch-Ua), one more small difference from a real browser that headers alone can't fix.
What About fake-useragent?
The fake-useragent package gives you a ready-made list of user agents, and it's handy for quick experiments:
from fake_useragent import UserAgent
ua = UserAgent(browsers=["Chrome", "Firefox"], os=["Windows", "Mac OS X", "Linux"], platforms=["desktop"])
print(ua.random)Treat it with care. When I tested version 2.2.0:
The default is mostly mobile. Mobile devices make up about 69% of its weighted data; in 3,000 draws from plain
UserAgent().random, 2,070 were mobile, including in-app browsers such as the Google and Facebook apps. Always passplatforms=["desktop"]unless you want mobile pages.The data is old. Its newest Chrome is 136 and its newest Firefox 138, both from spring 2025, while real browsers were at 157 and 160. It also contains Chrome versions as old as 65.
Failures are easy to miss. Asking for a
min_versionit has no data for doesn't raise an exception. It logs a short error and returns a fallback Edge 122 user agent.
And a user agent alone is only one line of a profile. Drop one of its strings into a profile and the linter shows the problem:
from fake_useragent import UserAgent
from profiles import CHROME_WINDOWS, check_profile
ua = UserAgent(browsers=["Chrome"], os=["Windows"], platforms=["desktop"])
mixed = {**CHROME_WINDOWS, "User-Agent": ua.random}
print(check_profile(mixed))
# ['user agent says Chrome 135 but sec-ch-ua disagrees'] (the version varies)Your own captured profiles, refreshed every few months, beat a random list.
The Accept-Encoding Trap
This is the one header you shouldn't copy from your browser. Chrome advertises gzip, deflate, br, zstd. If you paste that into a requests setup that doesn't have the brotli package installed, the server may answer in Brotli and response.text comes back as binary garbage. I reproduced exactly this. Leave Accept-Encoding to the library, as the code above does, or install brotli and zstandard first.
Beyond Headers
Headers are only one signal. Sites can also look at the TLS handshake your client performs, HTTP/2 details, the order and capitalization of headers, your IP reputation and whether you run their JavaScript. Python's requests can't change most of that, and no header profile can hide it. Some libraries address the lower layers. curl_cffi, for example, impersonates a browser's connection and sends a matching, complete header set:
from curl_cffi import requests as cffi_requests
response = cffi_requests.get("http://127.0.0.1:8000", impersonate="chrome")
print(response.json()["user-agent"])If a site has gone to the trouble of blocking automated traffic, more disguise only starts an arms race.
Using This Responsibly
Header rotation is a tool for being compatible, not for being invisible.
Prefer honesty where you can. A descriptive agent such as
MyBot/1.0 (+https://example.com/bot; [email protected])lets site owners contact you instead of blocking you. Use it unless a site rejects it for no good reason.Never rotate to dodge a rule.
robots.txtrules are addressed to crawlers by name, and a*group applies to everyone. Changing your identity to slip past aDisallow, a rate limit or a ban you were given is circumvention, and it can turn a terms-of-service problem into a bigger one. Our guide "Is Web Scraping Legal?" explains why.Don't raise your volume. Rotation doesn't change how many requests you send. Keep your delays and respect
429responses.Legitimate uses exist. Making a permitted scraper work with a site that rejects library agents, or testing how your own site responds to different browsers.
Troubleshooting
Symptom | Likely cause | What to try |
|---|---|---|
| The block uses your IP, TLS fingerprint or a JavaScript check, not your headers | Slow down and re-read the site's rules before trying anything more |
| You advertised | Drop |
Parser breaks on some requests | You're getting mobile and desktop HTML at random | Use desktop-only profiles |
CAPTCHAs appear after rotating | Identity changes mid-session, or one IP shows many browsers | Rotate per session and pair each identity with one proxy |
| Header set was mixed from different sources | Recapture the whole set from one browser |
Scrapy ignores your | The middleware overwrites it | That's intended; edit the profiles instead |
Where to Go from Here
You now have a pattern you can trust: capture real header sets, lint them, rotate them per session, and check what actually leaves your machine with the echo server. From here:
Combine profiles with rotating proxies and a polite delay (see our guides on both).
If a site needs JavaScript, use a real browser instead of imitating one (see our Playwright guide).
Happy scraping, and please do it politely!
Found this helpful? Share it with a fellow developer or leave your questions in the comments.
Comments(0)
Join the discussion by logging into your account.
No comments yet. Be the first to comment!