Scraping

Rotating User Agents and Headers in Python: A Practical Guide

Learn how to rotate User-Agent strings and HTTP headers in Python using requests, httpx, and Scrapy with consistent browser profiles.

·
9 min read
·
AI-assisted

AI tooling disclosure

Generative AI materially contributed to this article.

Human review confirmed by the author.

Rotating User Agents and Headers in Python: A Practical Guide

Every request your scraper sends carries a small introduction: who am I, and what can I handle? Left alone, a Python script introduces itself as python-requests/2.34.2, and some sites reject that on sight, even when they would happily serve a polite scraper.

The usual advice is to "rotate your user agent", and the usual code picks a random string from a list. That is a start, but on its own it is often worse than doing nothing. A Chrome user agent that arrives without the other headers Chrome sends is a client that doesn't exist in the real world. This guide shows a better approach: rotate complete, consistent header profiles, per session rather than per request, and check what you actually send.

Version note: tested with Python 3.13, requests 2.34, httpx 0.28, Scrapy 2.20, curl_cffi 0.17 and fake-useragent 2.2. The browser versions in the code (Chrome 152, Firefox 155) are a few releases behind the current ones (Chrome 157, with 158 rolling out, and Firefox 160 in mid-November 2026). Browsers update every few weeks, so treat the versions as placeholders to refresh.


What the Server Sees

You can't improve what you can't see, so start by building a tiny server that echoes back the headers it receives. Save this as echo_server.py:

import json
from http.server import BaseHTTPRequestHandler, HTTPServer


class Echo(BaseHTTPRequestHandler):
    def do_GET(self):
        body = json.dumps(dict(self.headers.items()), indent=2).encode()
        self.send_response(200)
        self.send_header("Content-Type", "application/json")
        self.send_header("Content-Length", str(len(body)))
        self.end_headers()
        self.wfile.write(body)

    def log_message(self, *args):    # keep the terminal quiet
        pass


HTTPServer(("127.0.0.1", 8000), Echo).serve_forever()

Run it in one terminal with python echo_server.py, install the libraries in another (pip install requests httpx), and look at what a plain requests call sends:

import requests

print(requests.get("http://127.0.0.1:8000", timeout=10).text)
{
  "Host": "127.0.0.1:8000",
  "User-Agent": "python-requests/2.34.2",
  "Accept-Encoding": "gzip, deflate, br, zstd",
  "Accept": "*/*",
  "Connection": "keep-alive"
}

(Your Accept-Encoding depends on which compression libraries are installed. That detail matters later.) Compare this with what Chrome sends for an ordinary page visit: a dozen headers, including language preferences, a detailed Accept list and several sec- headers. A server that looks at all of them sees an obvious difference.

What a Browser Sends

You don't need to memorize every header, but you should know the ones that matter:

Header

What it says

Notes

User-Agent

Browser, major version and OS family

Chrome freezes the minor version (152.0.0.0) and Safari freezes the macOS version (10_15_7), so little else varies

Accept

Content types the client can handle

Browsers send a long, browser-specific list; libraries send */*

Accept-Language

Preferred languages

Libraries often send none at all

Accept-Encoding

Compression the client can decode

Must match what your library can actually decompress

sec-ch-ua, sec-ch-ua-mobile, sec-ch-ua-platform

Client Hints repeating the browser, device type and OS

Sent by Chrome and other Chromium browsers, not by Firefox or Safari

Sec-Fetch-Site, -Mode, -Dest, -User

How the request was triggered

none for a typed URL, same-origin for a click within a site

Referer

The page you came from

Absent on a first visit

The important idea is consistency. The user agent, the Client Hints, the platform and the browser version must all tell the same story.

Level 1: Rotate the User-Agent

The simplest version picks a user agent at random for each request:

import random

import requests

USER_AGENTS = [
    "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/152.0.0.0 Safari/537.36",
    "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/152.0.0.0 Safari/537.36",
    "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:155.0) Gecko/20100101 Firefox/155.0",
]


def get(url):
    headers = {"User-Agent": random.choice(USER_AGENTS)}
    return requests.get(url, headers=headers, timeout=10)


for _ in range(3):
    print(get("http://127.0.0.1:8000").json()["User-Agent"])

This works as far as it goes, and a lenient site may need nothing more. But look at what else arrives: Accept: */*, no Accept-Language, no sec-ch-ua. Every one of those requests claims to be a modern browser while sending the headers of a Python library.

Level 2: Rotate Whole Header Profiles

A profile is a complete header set copied from a real browser. To capture one, open your browser's developer tools, go to the Network tab, load a page, right-click the first request and choose Copy as cURL. Keep headers like the ones in the profile below, and drop Cookie, Host and Accept-Encoding, which your library should manage itself.

Create profiles.py:

import re

CHROME_WINDOWS = {
    "sec-ch-ua": '"Chromium";v="152", "Not?A_Brand";v="24", "Google Chrome";v="152"',
    "sec-ch-ua-mobile": "?0",
    "sec-ch-ua-platform": '"Windows"',
    "Upgrade-Insecure-Requests": "1",
    "User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/152.0.0.0 Safari/537.36",
    "Accept": "text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.7",
    "Sec-Fetch-Site": "none",
    "Sec-Fetch-Mode": "navigate",
    "Sec-Fetch-User": "?1",
    "Sec-Fetch-Dest": "document",
    "Accept-Language": "en-US,en;q=0.9",
}

CHROME_MACOS = {
    **CHROME_WINDOWS,
    "sec-ch-ua-platform": '"macOS"',
    "User-Agent": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/152.0.0.0 Safari/537.36",
}

PROFILES = [CHROME_WINDOWS, CHROME_MACOS]

These two are the headers that the Chrome 152 profile of curl_cffi (more on that library below) sent to my echo server, so use them as a starting point and replace them with a capture from your own browser when you can. Add Firefox or Safari profiles the same way. They have no sec-ch-ua headers, which is exactly why you copy them whole instead of building them by hand.

Hand-built or mixed-up profiles are easy to get wrong, so add a quick linter to the same file. It catches the mismatches that give a fake identity away:

def check_profile(headers):
    """Return a list of ways this header set contradicts itself."""
    h = {name.lower(): value for name, value in headers.items()}
    ua = h.get("user-agent", "")
    problems = []

    if "Chrome/" not in ua:
        if "sec-ch-ua" in h:
            problems.append("sec-ch-ua headers without a Chrome user agent")
        return problems

    if "sec-ch-ua" not in h:
        return ["Chrome user agent but no sec-ch-ua headers"]

    major = re.search(r"Chrome/(\d+)", ua).group(1)
    if f'v="{major}"' not in h["sec-ch-ua"]:
        problems.append(f"user agent says Chrome {major} but sec-ch-ua disagrees")

    platform = h.get("sec-ch-ua-platform", "").strip('"')
    expected = "Windows" if "Windows" in ua else "macOS" if "Macintosh" in ua else "Linux"
    if platform != expected:
        problems.append(f"user agent says {expected} but sec-ch-ua-platform says {platform or 'nothing'}")
    return problems


for profile in PROFILES:
    assert check_profile(profile) == []

Level 3: Rotate Sessions, Not Requests

A real visitor doesn't change browsers between page views. If you rotate on every request, one cookie-carrying session appears to switch from Windows Chrome to a Mac a second later. Rotate per session instead: choose a profile, keep it for a few dozen requests, then start over with a fresh identity and fresh cookies. Save this as browser_session.py next to profiles.py:

import random
from urllib.parse import urlsplit

import requests

from profiles import PROFILES


class BrowserSession:
    """A requests session that behaves like one consistent browser."""

    def __init__(self, profiles=PROFILES, rotate_after=(20, 40)):
        self.profiles = profiles
        self.rotate_after = rotate_after
        self._new_identity()

    def _new_identity(self):
        self.profile = random.choice(self.profiles)
        self.session = requests.Session()               # a new session means new cookies
        self.session.headers.update(self.profile)       # replaces User-Agent and Accept
        self.limit = random.randint(*self.rotate_after)
        self.count = 0
        self.last_url = None

    def get(self, url, **kwargs):
        if self.count >= self.limit:
            self._new_identity()

        headers = {}
        if self.last_url:                               # behave like a click from the last page
            same_site = urlsplit(url).netloc == urlsplit(self.last_url).netloc
            headers["Referer"] = self.last_url
            headers["Sec-Fetch-Site"] = "same-origin" if same_site else "cross-site"

        kwargs.setdefault("timeout", 10)
        response = self.session.get(url, headers=headers, **kwargs)
        self.count += 1
        self.last_url = url
        return response


if __name__ == "__main__":
    browser = BrowserSession()
    for page in range(1, 4):
        reply = browser.get(f"http://127.0.0.1:8000/page-{page}").json()
        print(reply["User-Agent"][:60], "| Referer:", reply.get("Referer"))

Three details are worth noticing:

  • session.headers.update(profile) overwrites the library's User-Agent and Accept and adds everything else. It leaves Accept-Encoding to requests for a reason covered below.

  • The first request has no Referer and Sec-Fetch-Site: none, like a typed address. Later requests look like clicks.

  • A new Session is a new identity. Cookies don't leak from one "visitor" to the next.

If you also use proxies (see our guide to rotating proxies in Python), give each identity its own proxy and keep the pair together. Five different browsers behind one IP address looks just as odd as one browser jumping between five IP addresses mid-session.

The Same Idea in httpx

The profile dictionaries work unchanged with httpx:

import random

import httpx

from profiles import PROFILES

profile = random.choice(PROFILES)

with httpx.Client(headers=profile, timeout=10) as client:
    first = client.get("http://127.0.0.1:8000/page-1")
    print(first.json()["User-Agent"])

httpx keeps its own defaults (Connection, Accept-Encoding) and replaces User-Agent and Accept with yours. Like requests, it sends headers in its own order rather than a browser's.

Rotating Profiles in Scrapy

In Scrapy, a downloader middleware is the right place. Put profiles.py inside your project package (here it's called myproject), then create the middleware in myproject/middlewares.py:

import random

from myproject.profiles import PROFILES


class RotateProfileMiddleware:
    """Give every request a coherent browser header set."""

    def process_request(self, request):
        profile = request.meta.setdefault("profile", random.choice(PROFILES))
        for name, value in profile.items():
            request.headers[name] = value

Enable it in settings.py:

DOWNLOADER_MIDDLEWARES = {
    "myproject.middlewares.RotateProfileMiddleware": 543,
}

The number matters: 543 runs after Scrapy's built-in header middlewares (at 400 and 500), so your values replace their defaults. This also overrides the USER_AGENT setting. Because the chosen profile is stored in request.meta["profile"], a retried or redirected request keeps the same identity. New requests created with response.follow() don't inherit meta, so to pin a whole chain of requests to one identity, pass it along explicitly:

yield response.follow(url, callback=self.parse_page, meta={"profile": response.meta["profile"]})

Scrapy normalizes header names (sec-ch-ua goes out as Sec-Ch-Ua), one more small difference from a real browser that headers alone can't fix.

What About fake-useragent?

The fake-useragent package gives you a ready-made list of user agents, and it's handy for quick experiments:

from fake_useragent import UserAgent

ua = UserAgent(browsers=["Chrome", "Firefox"], os=["Windows", "Mac OS X", "Linux"], platforms=["desktop"])
print(ua.random)

Treat it with care. When I tested version 2.2.0:

  • The default is mostly mobile. Mobile devices make up about 69% of its weighted data; in 3,000 draws from plain UserAgent().random, 2,070 were mobile, including in-app browsers such as the Google and Facebook apps. Always pass platforms=["desktop"] unless you want mobile pages.

  • The data is old. Its newest Chrome is 136 and its newest Firefox 138, both from spring 2025, while real browsers were at 157 and 160. It also contains Chrome versions as old as 65.

  • Failures are easy to miss. Asking for a min_version it has no data for doesn't raise an exception. It logs a short error and returns a fallback Edge 122 user agent.

And a user agent alone is only one line of a profile. Drop one of its strings into a profile and the linter shows the problem:

from fake_useragent import UserAgent

from profiles import CHROME_WINDOWS, check_profile

ua = UserAgent(browsers=["Chrome"], os=["Windows"], platforms=["desktop"])
mixed = {**CHROME_WINDOWS, "User-Agent": ua.random}
print(check_profile(mixed))
# ['user agent says Chrome 135 but sec-ch-ua disagrees']   (the version varies)

Your own captured profiles, refreshed every few months, beat a random list.

The Accept-Encoding Trap

This is the one header you shouldn't copy from your browser. Chrome advertises gzip, deflate, br, zstd. If you paste that into a requests setup that doesn't have the brotli package installed, the server may answer in Brotli and response.text comes back as binary garbage. I reproduced exactly this. Leave Accept-Encoding to the library, as the code above does, or install brotli and zstandard first.

Beyond Headers

Headers are only one signal. Sites can also look at the TLS handshake your client performs, HTTP/2 details, the order and capitalization of headers, your IP reputation and whether you run their JavaScript. Python's requests can't change most of that, and no header profile can hide it. Some libraries address the lower layers. curl_cffi, for example, impersonates a browser's connection and sends a matching, complete header set:

from curl_cffi import requests as cffi_requests

response = cffi_requests.get("http://127.0.0.1:8000", impersonate="chrome")
print(response.json()["user-agent"])

If a site has gone to the trouble of blocking automated traffic, more disguise only starts an arms race.

Using This Responsibly

Header rotation is a tool for being compatible, not for being invisible.

  • Prefer honesty where you can. A descriptive agent such as MyBot/1.0 (+https://example.com/bot; [email protected]) lets site owners contact you instead of blocking you. Use it unless a site rejects it for no good reason.

  • Never rotate to dodge a rule. robots.txt rules are addressed to crawlers by name, and a * group applies to everyone. Changing your identity to slip past a Disallow, a rate limit or a ban you were given is circumvention, and it can turn a terms-of-service problem into a bigger one. Our guide "Is Web Scraping Legal?" explains why.

  • Don't raise your volume. Rotation doesn't change how many requests you send. Keep your delays and respect 429 responses.

  • Legitimate uses exist. Making a permitted scraper work with a site that rejects library agents, or testing how your own site responds to different browsers.

Troubleshooting

Symptom

Likely cause

What to try

403 even with a full browser profile

The block uses your IP, TLS fingerprint or a JavaScript check, not your headers

Slow down and re-read the site's rules before trying anything more

response.text is binary garbage

You advertised br or zstd without the matching package

Drop Accept-Encoding or install brotli and zstandard

Parser breaks on some requests

You're getting mobile and desktop HTML at random

Use desktop-only profiles

CAPTCHAs appear after rotating

Identity changes mid-session, or one IP shows many browsers

Rotate per session and pair each identity with one proxy

check_profile reports a mismatch

Header set was mixed from different sources

Recapture the whole set from one browser

Scrapy ignores your USER_AGENT setting

The middleware overwrites it

That's intended; edit the profiles instead

Where to Go from Here

You now have a pattern you can trust: capture real header sets, lint them, rotate them per session, and check what actually leaves your machine with the echo server. From here:

  • Combine profiles with rotating proxies and a polite delay (see our guides on both).

  • If a site needs JavaScript, use a real browser instead of imitating one (see our Playwright guide).

Happy scraping, and please do it politely!


Found this helpful? Share it with a fellow developer or leave your questions in the comments.

Comments(0)

Join the discussion by logging into your account.

No comments yet. Be the first to comment!

Bhavya Arora

Passionate developer sharing knowledge about modern web technologies and best practices.

Subscribe to Bhavya Arora's Newsletter

Direct email dispatches when new stories are published. Zero algorithms.