Rate Limiting and Human-Like Delays

Step 1: Establish a Baseline Request Rate

Every website has a different tolerance for request volume. A major e-commerce platform handles millions of pageviews per day and may not notice 100 requests per minute from a single IP. A small business website running on shared hosting might flag 10 requests per minute as suspicious. You need to find the right rate for each target.

Start conservatively at one request every 5 seconds and gradually increase speed while monitoring for warning signs: HTTP 429 (Too Many Requests) responses, 503 errors, CAPTCHA challenges, or responses that return different content than expected (some sites serve simplified or empty pages to suspected bots).

Check the target site's robots.txt file for a Crawl-delay directive. While not all crawlers honor this, and it is not part of the original robots.txt standard, many site operators use it to signal their preferred crawl rate. A Crawl-delay: 10 tells you the site prefers at most one request every 10 seconds from a single crawler.

For API endpoints, check the response headers for rate limit information. Many APIs include headers like X-RateLimit-Limit, X-RateLimit-Remaining, and X-RateLimit-Reset that tell you exactly how many requests you can make and when the limit resets. Respecting these limits prevents blocks entirely and makes your scraping sustainable.

Document your findings per domain. Maintain a configuration file or database that records the safe request rate for each target site, updated based on your ongoing monitoring. This saves you from rediscovering limits when you revisit a site weeks or months later.

Step 2: Add Randomized Delay Jitter

Fixed delays create perfectly regular request patterns that are trivially detectable. If your server logs show requests arriving exactly every 3.000 seconds, that is obviously automated. Real browsing has natural variation: sometimes a user clicks quickly (500ms between requests), sometimes they pause to read (15 seconds or more).

The simplest approach is uniform random jitter: instead of waiting exactly 3 seconds, wait a random time between 1.5 and 6 seconds. This eliminates the mechanical regularity of fixed delays. In Python: time.sleep(random.uniform(1.5, 6.0)). In JavaScript: await new Promise(r => setTimeout(r, 1500 + Math.random() * 4500)).

A more realistic distribution uses a log-normal or gamma distribution, which produces mostly short delays with occasional longer pauses. This matches the way humans actually browse: quick clicks through a list punctuated by longer reading pauses. In Python, random.lognormvariate(1.0, 0.5) generates values concentrated around 2.7 seconds with a long tail reaching 10 seconds or more.

For pagination crawling, add extra-long pauses every 5 to 15 pages to simulate a user taking a break, scrolling back up, or getting distracted. Insert a 15 to 45 second pause at random intervals to break up the otherwise steady cadence of sequential page loads.

Avoid patterns in your randomization. Some implementations use the same random seed across sessions, producing the same "random" sequence every time. Use truly random delays by seeding your random number generator from the system clock or using a cryptographic random source.

Step 3: Implement Exponential Backoff

Exponential backoff is your automatic response to rate limit signals. When a request fails with a rate-related error, you increase the delay before retrying. Each consecutive failure doubles the delay, preventing your scraper from worsening the situation.

A standard implementation works like this: the first retry waits 2 seconds, the second waits 4 seconds, the third waits 8 seconds, the fourth waits 16 seconds, and so on up to a maximum cap (usually 60 to 300 seconds). Add random jitter to each backoff interval to prevent synchronized retry storms when running multiple concurrent scrapers.

The formula for backoff with jitter is: delay = min(base * 2^attempt + random(0, base), max_delay). With a base of 2 seconds and a max of 120 seconds, the delays would be approximately 2, 4, 8, 16, 32, 64, 120, 120 seconds (plus jitter at each step).

Reset the backoff counter after a successful request. If you receive a successful response after backing off, return to your normal request rate rather than staying at the elevated delay. But be cautious: some anti-bot systems temporarily allow requests after rate limiting and then block again if the same aggressive rate resumes. Gradually ramp back up to your normal speed over several successful requests rather than immediately returning to full speed.

If a 429 response includes a Retry-After header, always respect it. This header specifies either a number of seconds to wait or a date-time when you may retry. Using the server's explicit guidance is always better than guessing with exponential backoff.

Step 4: Simulate Page Reading Time

Human visitors spend different amounts of time on different types of pages. A search results page gets a quick scan (3 to 8 seconds) before clicking through. A detailed product page or article gets a longer read (15 to 60 seconds or more). A form page takes time to fill out (30 to 120 seconds). Your scraper should reflect these patterns.

After scraping a content-heavy page (detected by word count, page size, or URL pattern), add a longer delay before the next request. For product detail pages, pause 5 to 15 seconds. For article pages, pause 10 to 30 seconds. For listing or category pages, a normal 2 to 5 second delay is appropriate.

If your scraper follows links from a listing page to detail pages and back, the timing between "click through to detail" and "return to listing" should reflect realistic reading behavior. A pattern of: listing page (2 second pause), detail page 1 (12 second pause), detail page 2 (8 second pause), detail page 3 (18 second pause), next listing page (3 second pause), is much more realistic than the same 2-second delay for every transition.

For sites that track scroll behavior via JavaScript, a headless browser can simulate realistic scrolling patterns. Scroll down gradually at varying speeds, pause at different scroll positions, and occasionally scroll back up. Libraries like Playwright provide smooth scrolling APIs that make this straightforward to implement.

Step 5: Schedule Scraping During Peak Traffic

Websites experience predictable traffic patterns that follow their audience's daily routines. A US-focused e-commerce site sees peak traffic between 10 AM and 9 PM Eastern time, with the highest volume in the early afternoon. A European news site peaks during European business hours. A B2B SaaS platform is busiest on weekday mornings.

Scraping during peak hours offers two advantages. First, your requests blend into a much larger volume of legitimate traffic, making them harder to identify through volume analysis alone. Second, the site's infrastructure is scaled to handle peak load, so your additional requests are a smaller percentage of total traffic and less likely to cause performance issues.

Scraping during off-peak hours (midnight to 6 AM local time) is risky because your traffic constitutes a larger share of total volume and stands out more clearly in traffic analysis. Some sites reduce their rate limits during off-peak hours precisely because they expect lower legitimate traffic.

For sites with global audiences, there is no true off-peak period, so this consideration matters less. But for regionally focused sites, aligning your scraping schedule with their peak hours is a simple, effective strategy.

If your scraping job spans multiple days, vary your daily schedule slightly. Running at exactly 10:00 AM every day is a detectable pattern. Starting between 9:30 and 10:30 with a random offset each day looks more natural.

Step 6: Distribute Requests Across Endpoints

Concentrated request patterns are easier to detect than distributed ones. If all your requests target /products/page/1 through /products/page/500 in sequence, the pattern is obvious even at a slow rate. Distributing requests across different sections of the site makes your traffic look more like organic browsing.

Instead of scraping all products sequentially, interleave requests across categories: one request to electronics, one to clothing, one to home goods, then back to electronics. This distributes your load across different application servers (many large sites route different categories to different backend systems) and creates a browsing pattern that resembles a real user exploring multiple sections.

Include ancillary requests that real browsers would make. Load the homepage occasionally. Visit the about page or help center. Follow internal links from pages you scrape. This creates a more natural browsing session with diverse URL patterns rather than a single-purpose crawl focused exclusively on data pages.

For large-scale scraping jobs, use multiple concurrent sessions with different entry points and crawl orders. One session starts from category A and works forward; another starts from category Z and works backward; a third follows the site's internal link structure organically. The combined traffic pattern across all sessions looks much more diverse than a single systematic crawl.