Rotating User Agents and Headers

Step 1: Build a Current User-Agent List

Your User-Agent list should contain only strings from current, widely-used browser versions. Using outdated or rare User-Agents makes your traffic stand out in server logs. As of mid-2026, the most common desktop browsers are Chrome 126 through 128, Firefox 128 through 130, Safari 18, and Edge 126 through 128 (which shares Chrome's rendering engine).

Each User-Agent string encodes the browser name, version, rendering engine, and operating system. A real Chrome User-Agent on Windows looks like: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/127.0.0.0 Safari/537.36. The same browser on macOS changes the platform token: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/127.0.0.0 Safari/537.36.

Build your list from real browser traffic data rather than generating strings manually. Sources include browser version statistics from StatCounter or similar analytics services, and the User-Agent strings your own browser sends (check via developer tools under Network headers). Maintain lists of 20 to 50 current User-Agents covering the top browsers on Windows, macOS, and Linux. Update this list monthly as new browser versions are released.

Avoid these common mistakes: using User-Agent strings from 2020-era browsers, including obviously fake or malformed strings, mixing mobile and desktop User-Agents without matching the rest of the request profile, and using User-Agents for browsers that have been discontinued.

Step 2: Match Headers to Each User-Agent

A User-Agent string is just one header in a request. Modern browsers send 15 to 20 headers with each navigation request, and many of these headers are browser-specific. The key is to group your User-Agent with all the other headers that browser actually sends, creating a complete "browser profile."

For Chrome-based browsers, the critical companion headers are the Client Hints family. The sec-ch-ua header lists the browser brands and versions: "Chromium";v="127", "Not)A;Brand";v="99", "Google Chrome";v="127". The sec-ch-ua-mobile header is ?0 for desktop. The sec-ch-ua-platform header must match the OS in the User-Agent: "Windows", "macOS", or "Linux".

The Sec-Fetch headers describe the request context. For a top-level page navigation: Sec-Fetch-Dest: document, Sec-Fetch-Mode: navigate, Sec-Fetch-Site: none (for direct navigation) or same-origin (for internal links), and Sec-Fetch-User: ?1. These values change for sub-resource requests like images, scripts, and AJAX calls.

The Accept header varies by browser. Chrome sends text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.7 for HTML pages. Firefox uses a slightly different format. Safari differs further. Always pair the correct Accept value with each browser identity.

Accept-Language should match the locale implied by your proxy geography and User-Agent OS locale. A US residential proxy should send en-US,en;q=0.9. A German proxy should send de-DE,de;q=0.9,en-US;q=0.8,en;q=0.7 or similar.

Create a data structure (JSON file or dictionary) that maps each User-Agent to its complete header set. When you select a User-Agent for a session, pull the entire profile and apply all headers together.

Step 3: Rotate Header Profiles Per Session

The key principle is: rotate profiles between sessions, but keep them consistent within a single session. A real user does not change browsers mid-session. If you send a Chrome User-Agent for the first request and a Firefox User-Agent for the second request using the same cookies and session ID, the server knows something is wrong.

At the start of each scraping session, select one browser profile randomly from your pool. Apply its complete header set to every request in that session. When the session ends (either because you finished the task, changed proxy IPs, or hit a time limit), start a new session with a different randomly-selected profile.

Weight your random selection toward the most popular browsers. Chrome holds roughly 65 percent market share, Safari about 18 percent, Firefox about 7 percent, and Edge about 5 percent. Your rotation should approximately reflect these proportions, so about two-thirds of your sessions use Chrome profiles.

If you are running multiple concurrent scraping sessions, ensure that each session uses a different profile. Two simultaneous sessions from the same IP range with identical Chrome 127 Windows User-Agents look suspicious, while one Chrome session and one Firefox session look like two different users on the same network.

Step 4: Preserve Correct Header Order

Header order is an underappreciated signal. Real browsers send headers in a specific, consistent order that differs between browser families. Anti-bot systems fingerprint this order because most HTTP libraries do not replicate it correctly.

Chrome sends headers in this approximate order: Host, Connection, sec-ch-ua, sec-ch-ua-mobile, sec-ch-ua-platform, Upgrade-Insecure-Requests, User-Agent, Accept, Sec-Fetch-Site, Sec-Fetch-Mode, Sec-Fetch-User, Sec-Fetch-Dest, Accept-Encoding, Accept-Language, Cookie. Firefox uses a different order, placing User-Agent and Accept before the security headers.

Python's requests library sends headers in the order you add them to the dictionary (as of Python 3.7+, dictionaries maintain insertion order). Use an OrderedDict or simply add headers in the correct browser-specific sequence. In Node.js, most HTTP libraries also preserve insertion order.

The simplest approach is to define your header profiles as ordered lists of key-value pairs rather than unordered dictionaries. When applying a profile, iterate through the list and add each header in sequence. This guarantees the correct order regardless of which programming language or HTTP library you use.

If you use a library that sorts headers alphabetically or otherwise reorders them, switch to a library that preserves order, or use a lower-level HTTP client where you control the raw request construction.

Step 5: Match TLS Fingerprint to Browser Identity

Every TLS client produces a unique fingerprint during the handshake process. The JA3 fingerprinting method (and its successor JA4) hashes the cipher suites, extensions, elliptic curves, and other parameters that a client advertises when establishing a TLS connection. Each browser version produces a specific JA3 hash, and so does each HTTP library.

If your User-Agent claims to be Chrome 127 but your TLS handshake produces the JA3 hash of Python's urllib3, the mismatch is a dead giveaway. Anti-bot services like Cloudflare and Akamai check JA3/JA4 fingerprints as standard practice.

The most reliable solution is to use a real browser via Playwright, Puppeteer, or Selenium. The browser engine handles TLS natively, producing the correct fingerprint automatically. For HTTP-level scraping without a browser, use impersonation libraries: curl-impersonate patches curl to mimic browser TLS fingerprints, and libraries like tls-client (Go) and cycletls (Node.js) provide similar functionality.

In Python, the curl_cffi library wraps curl-impersonate and supports direct browser impersonation. You specify the target browser (e.g., impersonate="chrome127") and the library handles TLS fingerprinting, HTTP/2 settings, and header ordering automatically. This is currently the most practical solution for Python scrapers that need TLS fingerprint matching without the overhead of running a full browser.

Step 6: Vary Headers by Request Type

Real browsers do not send identical headers for every request. A page navigation, an AJAX call, an image load, and a script fetch each send different header combinations. If your scraper sends navigation-style headers for an API endpoint, or AJAX-style headers for a page load, the inconsistency is detectable.

For page navigations (loading a new HTML page), use full navigation headers: Sec-Fetch-Dest: document, Sec-Fetch-Mode: navigate, and the complete Accept string for HTML content.

For AJAX/XHR requests, change the headers accordingly: Sec-Fetch-Dest: empty, Sec-Fetch-Mode: cors, Accept: application/json (or whatever the API expects), and add X-Requested-With: XMLHttpRequest if the site expects it.

For image loads: Sec-Fetch-Dest: image, Sec-Fetch-Mode: no-cors, Accept: image/avif,image/webp,image/apng,image/svg+xml,image/*,*/*;q=0.8.

When scraping a JavaScript-rendered site with a headless browser, the browser handles all of this automatically. But when making direct HTTP requests, you need to set the appropriate headers for each request type manually. This is particularly important when scraping APIs that serve data to frontend JavaScript, as these endpoints expect AJAX-style headers and will reject requests with navigation headers.