How to Handle Cloudflare and Anti-Bot Walls

Step 1: Identify Which Anti-Bot Service Protects the Site

Before attempting any bypass, you need to know what you are dealing with. Each anti-bot service leaves distinctive fingerprints in response headers, cookies, and loaded JavaScript files.

Cloudflare is identified by the cf-ray and cf-cache-status response headers, the __cf_bm and cf_clearance cookies, and challenge pages served from challenges.cloudflare.com. The server: cloudflare header appears in most responses. When a challenge is triggered, you see a 403 response with an HTML page that loads JavaScript from Cloudflare's CDN.

DataDome sets a datadome cookie and includes a JavaScript tag from js.datadome.co. Challenge pages redirect to geo.captcha-delivery.com. The response header x-datadome appears on protected requests.

PerimeterX (HUMAN) loads JavaScript from domains containing px-cdn.net or px-client.net. It sets cookies prefixed with _px and uses a sensor script that collects behavioral data. Challenge pages contain PerimeterX branding.

Akamai Bot Manager sets the _abck cookie and loads a sensor script from the target domain (not from an external CDN). The ak_bmsc cookie is also common. Akamai challenges are served inline rather than as redirects.

You can identify the service using a simple curl request: curl -I https://target-site.com. The response headers usually reveal the CDN and anti-bot service. For JavaScript-based detection, open the site in a regular browser with DevTools and check the Network tab for external script loads from the providers listed above.

Step 2: Configure a Stealth Headless Browser

A stealth-configured headless browser is the foundation for handling any anti-bot service. The browser executes JavaScript challenges natively, produces correct TLS fingerprints, and generates realistic canvas and WebGL fingerprints.

Playwright with the playwright-extra and puppeteer-extra-plugin-stealth packages is the most common setup. The stealth plugin applies patches that address the most well-known detection vectors: setting navigator.webdriver to undefined, spoofing chrome.runtime, adding realistic plugin arrays, patching the permissions API, spoofing WebGL vendor and renderer strings, and modifying various other properties that differ between headless and headed browsers.

Configure the browser with realistic parameters. Set the viewport to a common resolution like 1920x1080 or 1366x768. Set the locale and timezone to match your proxy's geography. Enable WebGL and GPU acceleration if possible, or use the correct software renderer strings for your server environment. Set a realistic list of languages in navigator.languages.

Launch the browser in headed mode when possible. Some anti-bot checks specifically test for headless mode by examining the User-Agent for "HeadlessChrome" or by checking navigator.plugins.length (which returns 0 in headless mode but 3 or more in headed mode). Running a headed browser on a server requires a virtual display (Xvfb on Linux), but it passes checks that headless mode fails.

Keep your browser version current. Anti-bot services maintain lists of known automation tool versions and their fingerprints. Running an outdated version of Playwright or Puppeteer uses browser binaries with known fingerprints that are specifically targeted. Update your tools at least monthly.

Step 3: Use Residential Proxies with Correct Geolocation

Anti-bot services check IP reputation as the first filter. Even a perfectly configured stealth browser will be challenged or blocked if it connects from a datacenter IP that is already flagged in the service's reputation database.

Use residential or ISP proxies that match the geographic region of the target site's audience. For a US e-commerce site, use US residential proxies. For a UK news site, use UK residential proxies. Some anti-bot services perform more granular checks, flagging requests that come from unusual regions for the type of content being accessed.

Configure your proxy at the browser level so that all traffic from the browser instance (including the challenge JavaScript's callbacks to the anti-bot service) routes through the same proxy. If you set the proxy only for your main requests but let challenge callbacks go through your server's direct connection, the anti-bot service sees two different IPs in the same session and flags it immediately.

Some residential proxy providers offer "premium" or "verified" pools with higher quality IPs that have been specifically tested against major anti-bot services. These pools are more expensive but have higher success rates because the IPs have clean reputations and are less likely to be flagged.

Step 4: Solve JavaScript Challenges and Preserve Cookies

When you first visit a Cloudflare-protected site, the server returns a challenge page with JavaScript that evaluates your browser. If the browser passes, Cloudflare sets the cf_clearance cookie, which grants access for subsequent requests. The challenge process is invisible to normal users (it takes less than a second) but requires full JavaScript execution.

With a stealth headless browser, the challenge usually resolves automatically. Navigate to the target URL, wait for the challenge to complete (detected by waiting for the URL to change from the challenge page to the actual content, or by waiting for the cf_clearance cookie to appear), and then proceed with your scraping.

The critical step is preserving the clearance cookies. After solving a challenge, extract the cookies from the browser context and save them. These cookies are valid for a period (Cloudflare's cf_clearance typically lasts 30 minutes to several hours, depending on the site's configuration). During their validity period, you can reuse them for subsequent requests without solving the challenge again.

If you need to make many requests quickly, solve the challenge once in a browser, extract the cookies and headers (including User-Agent and sec-ch-ua), and then use those cookies with a fast HTTP client like curl_cffi or httpx for the actual data collection. This hybrid approach gives you the speed of HTTP requests with the challenge-solving capability of a browser.

Be aware that clearance cookies are typically bound to the IP address that solved the challenge. Using a clearance cookie from a different IP address will trigger a new challenge. Match your cookies to your proxies: solve the challenge through proxy A, and only reuse those cookies through proxy A.

Step 5: Handle CAPTCHAs When They Appear

Even with good stealth configuration and residential proxies, some percentage of requests will trigger CAPTCHAs. Cloudflare Turnstile, hCaptcha, reCAPTCHA, and DataDome CAPTCHAs each require different handling approaches.

Cloudflare Turnstile is designed to be mostly invisible to real users, solving itself automatically in the background. In a stealth headless browser with good configuration, Turnstile often resolves without user interaction. If it does not, you need a CAPTCHA solving service that supports Turnstile tokens. Services like CapSolver and 2Captcha accept Turnstile site keys and return solution tokens.

hCaptcha requires selecting images matching a description. Automated solving services use human workers or AI models to solve these challenges. Send the site key and page URL to the service API, receive a solution token, and inject it into the page's response callback.

DataDome CAPTCHAs are served from captcha-delivery.com and have their own solving flow. After solving, the DataDome cookie is updated with a clearance value. Some CAPTCHA solving services have specific support for DataDome challenges.

Integrate CAPTCHA solving as a fallback, not as a primary strategy. If you are solving CAPTCHAs on more than 5 to 10 percent of requests, something in your stealth configuration is wrong and should be fixed rather than relying on CAPTCHA solving for every request. Each solve costs money (typically a fraction of a cent to a few cents each) and adds latency (5 to 30 seconds per solve).

Step 6: Maintain Session Integrity Under Anti-Bot Monitoring

Anti-bot services track sessions over time, not just individual requests. Consistency within a session is critical for avoiding mid-session detection and re-challenge.

Keep the following elements consistent within each scraping session: the proxy IP (never rotate mid-session), all HTTP headers including User-Agent and Client Hints, the browser fingerprint (canvas hash, WebGL renderer, font list), cookies (accept all cookies and send them back with each request), and the TLS fingerprint (use the same browser instance or HTTP client throughout).

Limit session length to mimic natural browsing. Sessions that last for hours with continuous activity are suspicious. End each session after 20 to 50 page loads, discard the cookies and browser context, rotate to a new proxy, and start a fresh session with a new browser fingerprint.

Monitor the target site's response behavior throughout the session. Watch for signs of progressive throttling: increasing response times, intermittent CAPTCHAs, or responses with different content structure. These signals indicate that the anti-bot system is growing suspicious of your session. End the session immediately when you see these signs rather than pushing through and risking a permanent block on your proxy IP.

For DataDome and PerimeterX specifically, be aware that their sensor scripts continuously collect behavioral data throughout the session. This includes mouse movements, scroll patterns, and click coordinates. If you are using a headless browser, simulate these events periodically: move the mouse to different positions, scroll the page, and interact with elements before extracting data. The absence of any mouse or scroll activity is a clear automation signal to these systems.