How to Scrape Without Getting Blocked

Assess the Target Before You Start

The amount of anti-detection effort you need depends entirely on how well the target site is protected. Over-engineering your setup for a site with no bot detection wastes time and money on proxies you do not need. Under-engineering for a Cloudflare-protected e-commerce site wastes time on failed requests and debugging.

Start by checking the target site with a simple curl request. If you get the full HTML content back with no challenge page, the site has minimal or no bot detection. If you get a 403, a CAPTCHA, or a JavaScript challenge page, note which anti-bot service is in use (check response headers for cloudflare, datadome, or perimeterx indicators) and plan accordingly.

Sites fall into roughly three protection tiers. Low protection sites (small business sites, blogs, public databases, government portals) can be scraped with basic HTTP requests and minimal precautions. Medium protection sites (mid-size e-commerce, news sites, some SaaS platforms) use Cloudflare or similar CDN-level protection and require residential proxies and proper headers. High protection sites (major e-commerce platforms, travel booking, social networks, ticketing) use advanced anti-bot services with behavioral analysis and require stealth headless browsers with full anti-detection configuration.

Layer 1: IP Management

Your IP address is the most basic signal that servers check, and it is the easiest to manage. The goal is to ensure that each IP you use has a clean reputation and that no single IP sends enough requests to trigger rate limits.

For low-protection sites, a small pool of datacenter proxies (10 to 50 IPs) with rotation is sufficient. For medium-protection sites, use residential proxies from a reputable provider with automatic rotation. For high-protection sites, use residential or mobile proxies with geo-targeting that matches the target site's audience location.

Implement per-IP request budgets: retire each IP after a fixed number of requests to the same domain (10 to 50 requests depending on protection level). Track error rates per IP and remove addresses that start returning errors or CAPTCHAs. Never use the same IP for multiple concurrent sessions on the same domain.

Layer 2: Request Headers

Headers tell the server what client is making the request. Every header must be consistent with the browser identity you are claiming.

Build complete browser profiles that include User-Agent, sec-ch-ua, sec-ch-ua-mobile, sec-ch-ua-platform, Accept, Accept-Language, Accept-Encoding, Sec-Fetch-Dest, Sec-Fetch-Mode, Sec-Fetch-Site, and Sec-Fetch-User. Use only current browser versions (Chrome 126+, Firefox 128+, Safari 18). Send headers in the correct browser-specific order.

Rotate profiles between sessions but keep each session's profile consistent across all its requests. Weight your rotation toward Chrome (65 percent of real traffic) with smaller proportions of Firefox, Safari, and Edge. Match Accept-Language to your proxy's geographic location.

Layer 3: TLS and Protocol Fingerprint

The TLS handshake fingerprint (JA3/JA4) and HTTP/2 settings reveal which client library is making the connection. A mismatch between your User-Agent and your TLS fingerprint is one of the strongest detection signals available.

For HTTP-level scraping, use a TLS-impersonating library: curl_cffi for Python, tls-client for Go, or cycletls for Node.js. These libraries reproduce the exact TLS handshake parameters and HTTP/2 settings of real browsers, making your connections indistinguishable at the protocol level.

For browser-based scraping, this layer is handled automatically because Playwright, Puppeteer, and Selenium use real browser engines that produce authentic TLS fingerprints. Make sure you are using the latest browser binaries, as older versions have known fingerprints that anti-bot services specifically target.

Layer 4: JavaScript Execution

Many modern websites require JavaScript execution to render content, and anti-bot services use JavaScript challenges to verify that a real browser is making the request.

If the target site requires JavaScript rendering (check by viewing the page source versus the rendered DOM in a browser), you need a headless browser. Configure it with stealth patches: playwright-extra with the stealth plugin for Playwright, puppeteer-extra-plugin-stealth for Puppeteer. These patches address navigator.webdriver, chrome.runtime, plugin arrays, WebGL renderer strings, and other common detection vectors.

For sites that do not require JavaScript (the page source contains the full content), you can use HTTP-level scraping with TLS impersonation. This is faster and uses less resources than running a browser, but it cannot pass JavaScript challenges if they are triggered.

A hybrid approach works well for many use cases: use a headless browser to solve initial challenges and obtain clearance cookies, then switch to fast HTTP requests using those cookies for the actual data collection. This gives you the challenge-solving capability of a browser with the speed and resource efficiency of direct HTTP requests.

Layer 5: Timing and Behavior

Realistic timing separates detectable scrapers from undetectable ones. Even with perfect proxies, headers, and TLS fingerprints, machine-speed request patterns are trivially identifiable.

Add randomized delays between requests using a log-normal distribution (concentrated around 2 to 4 seconds with a long tail reaching 10 to 15 seconds). Include longer pauses after content-heavy pages (10 to 20 seconds) to simulate reading time. Add periodic extended pauses (30 to 60 seconds) every 10 to 20 requests to simulate natural browsing breaks.

Schedule your scraping during the target site's peak traffic hours. Distribute requests across different sections of the site rather than scraping one section exhaustively before moving to the next. Include occasional requests to non-data pages (homepage, about page, help center) to create a more natural browsing session.

Implement exponential backoff for rate limit responses (429, 503). Double the delay after each consecutive failure up to a maximum cap. Reset backoff gradually after successful requests rather than immediately returning to full speed.

Layer 6: Session and Cookie Management

Cookies tie a session together and prove that your client is maintaining state like a real browser. Ignoring cookies is a clear automation signal.

Accept and return all cookies in every request. Use a persistent cookie jar (requests.Session in Python, or the browser's built-in cookie management in Playwright/Puppeteer). Match cookies to proxy IPs: never send cookies obtained through one IP from a different IP. Start fresh cookie sessions when rotating proxies.

Preserve challenge clearance cookies (like Cloudflare's cf_clearance) and reuse them for their full validity period. This avoids solving the same challenge repeatedly and reduces your CAPTCHA encounter rate.

Limit session length to 20 to 50 page loads. Longer sessions accumulate behavioral evidence that increases your detection risk. End each session cleanly, discard the cookies, rotate to a new proxy, and start fresh with a new browser profile.

Putting It All Together: A Decision Framework

When starting a new scraping project, follow this decision tree:

1. Can you get the data from an API or data feed? If the site offers a public API or data export, use it. APIs are faster, more reliable, and not subject to anti-bot measures. Check the site's developer documentation, robots.txt, and /api/ paths before resorting to scraping.

2. Does the site render content server-side? If the page source contains the full content (not just JavaScript bundle references), you can use HTTP-level scraping. This is faster and cheaper than running browsers.

3. What protection tier is the site? For low protection, use HTTP requests with proper headers and a small proxy pool. For medium protection, add TLS impersonation and residential proxies. For high protection, use stealth headless browsers with residential proxies and full behavioral simulation.

4. How much data do you need? Small jobs (hundreds to low thousands of pages) can afford slower request rates and do not require sophisticated proxy management. Large jobs (tens of thousands to millions of pages) need distributed infrastructure, robust proxy pools, and automated monitoring to maintain success rates over extended periods.

5. Is the data time-sensitive? If you need data daily (price monitoring, inventory tracking), invest in a robust, sustainable scraping setup that can run reliably over months. If you need a one-time data pull, you can tolerate more manual intervention and less automation.

Common Mistakes That Get You Blocked

Certain mistakes cause the majority of scraping blocks. Avoiding these alone will prevent most detection:

Using default library User-Agents. Python requests sends "python-requests/2.31.0" by default. Node's axios sends a similarly identifiable string. Always set a real browser User-Agent.

Ignoring cookies. Many scrapers discard Set-Cookie headers, which tells the server that the client is stateless and therefore automated. Always maintain a cookie jar.

Fixed timing. Requests arriving at perfectly regular intervals are an unmistakable automation signal. Always add randomized jitter to delays.

Sequential URL patterns. Scraping page-1, page-2, page-3 in order is the most obvious bot behavior. Randomize or interleave your request order.

Mismatched headers. A Chrome User-Agent with Firefox Accept headers, or a Windows User-Agent with sec-ch-ua-platform claiming "macOS", creates inconsistencies that servers check for. Use complete, consistent browser profiles.

Ignoring the Referer header. Real browsers send a Referer header that shows which page the user came from. Scrapers that omit this header or send the same Referer for every request look automated. Set the Referer to the previous page in your navigation sequence.

Not loading sub-resources. Real browsers automatically load CSS, JavaScript, images, and fonts referenced by each page. A client that only requests HTML documents is obviously not a browser. At minimum, make requests for the main CSS and JavaScript files to simulate sub-resource loading.