How to Crawl Large Websites

Step 1: Plan Your Crawl Scope

Before writing any code, define exactly what you need from the target site. Not every page on a large site is relevant to your use case. An SEO audit needs every indexable page but can skip login-required sections. A price monitor needs product pages but not blog posts or help articles. A research dataset might need only pages from specific date ranges or content categories.

Define your scope using URL patterns (which path prefixes to include or exclude), page types (identify pages by their template structure), depth limits (how many clicks from the homepage to follow), and content filters (skip pages that do not match your topic). A well-defined scope can reduce a 10-million-page site to a manageable subset of 100,000 relevant pages, dramatically reducing crawl time and storage requirements.

Also research the target site's structure before crawling. Check their sitemap.xml for a directory of all pages (many large sites publish comprehensive sitemaps). Review their robots.txt for disallowed paths and crawl-delay preferences. Browse the site manually to understand their URL patterns, pagination schemes, and navigation structure. This research prevents wasted crawl budget on irrelevant sections.

Step 2: Use Async I/O for Concurrency

Large-site crawling is fundamentally I/O bound. Each HTTP request takes 100-500ms of network round-trip time during which a synchronous crawler sits idle. Asynchronous I/O allows you to have hundreds of requests in flight simultaneously, utilizing the wait time from one request to process responses from others.

In Python, use asyncio with aiohttp for async HTTP requests. Create a semaphore to limit concurrent connections (typically 50-200 for a single domain, depending on the target server's capacity). Process responses as they arrive rather than waiting for batches. This pattern typically achieves 200-1000 pages per second from a single machine when crawling one domain, compared to 1-2 pages per second with synchronous requests.

For Scrapy users, concurrency is built-in via the CONCURRENT_REQUESTS setting (default 16, increase to 32-64 for large sites) and CONCURRENT_REQUESTS_PER_DOMAIN (default 8, increase cautiously while monitoring for rate limiting). Scrapy's Twisted reactor handles the async machinery transparently.

Monitor the target server's response times as you increase concurrency. If average response time increases significantly above baseline, you are pushing too hard and should reduce concurrent connections. The goal is maximum throughput without degrading the site's performance for other users.

Step 3: Build a Memory-Efficient Frontier

A large site crawl discovers millions of URLs that must be stored in the frontier (crawl queue). Keeping all URLs in a Python set or list quickly exhausts available RAM. A 10-million URL frontier with average 80-character URLs consumes roughly 2-4 GB of memory just for the strings, plus overhead for the data structure itself.

Several approaches solve this. Disk-backed queues like SQLite or LevelDB store URLs on disk with fast random access, trading some speed for virtually unlimited capacity. Redis provides a network-accessible queue that persists to disk and supports atomic operations for distributed crawlers. For deduplication specifically, bloom filters use fixed memory (regardless of set size) to test URL membership with a configurable false-positive rate; a bloom filter for 10 million URLs uses roughly 15 MB at a 1% false-positive rate.

Scrapy provides built-in disk-based request queues via the JOBDIR setting, which also enables pause/resume functionality. For custom crawlers, the pybloom-live or rbloom libraries provide memory-efficient bloom filter implementations.

Step 4: Implement Smart Deduplication

Large sites frequently serve the same content at multiple URLs. Session IDs in query parameters, tracking codes, sort order variations, pagination with different parameter formats, and print-friendly versions all create URL variants that lead to identical content. Without deduplication, your crawler wastes resources fetching and storing the same page dozens of times.

URL-level deduplication is the first defense: canonicalize URLs by removing known tracking parameters (utm_source, utm_medium, sessionid, etc.), sorting remaining query parameters alphabetically, removing default ports, lowercasing scheme and hostname, and normalizing path encoding. This eliminates the most common URL-level duplicates before any HTTP request is made.

Content-level deduplication catches cases where different URLs genuinely serve identical content. Compute a fingerprint (MD5 or SimHash) of the page's main content after stripping navigation, headers, and footers. Store fingerprints in a set or bloom filter. Skip pages whose content fingerprint matches a previously seen page. SimHash is particularly useful because it detects near-duplicates (pages with minor differences like personalized ads or timestamps) rather than requiring exact matches.

Step 5: Respect Rate Limits and Crawl Politely

Large sites have infrastructure designed to handle their normal traffic, but an aggressive crawler can represent a significant additional load. Getting blocked (via IP ban, CAPTCHA, or firewall rule) means losing all progress and potentially being permanently excluded. Politeness is both ethical and pragmatic.

Implement per-domain rate limiting with a configurable delay between requests. Start conservatively (1-2 seconds between requests) and gradually decrease the delay while monitoring for warning signs: increasing response times, 429 (Too Many Requests) status codes, 503 (Service Unavailable) responses, or CAPTCHA pages being served instead of content. Many large sites specify their preferred crawl rate in robots.txt via the Crawl-delay directive.

Rotate your requests across different sections of the site rather than exhaustively crawling one section before moving to the next. This distributes load across the target's infrastructure more evenly. Crawl during off-peak hours (typically overnight in the site's primary timezone) when server capacity is more available. Identify your crawler with a descriptive User-Agent string including a URL where the site operator can learn about your crawler and contact you.

Step 6: Handle Errors and Resume Gracefully

A large crawl running for hours or days will inevitably encounter errors: network timeouts, DNS failures, target server crashes, your own machine running out of disk space, or process crashes. Without checkpointing, you lose all progress and must restart from the beginning.

Implement periodic state checkpointing by saving the frontier queue, visited URL set, and crawl statistics to disk at regular intervals (every few minutes or every N pages). On restart, load the checkpoint and resume from where the crawler left off. Scrapy provides this via JOBDIR, which persists request queues and deduplication filters across runs.

For individual request failures, implement a retry queue with exponential backoff. Temporarily failed requests (timeouts, 503s, connection resets) go back into the queue with an increasing delay before retry. After a maximum number of retries (typically 3-5), mark the URL as permanently failed and log it for manual review. Track failure rates per domain to detect when a target site is experiencing problems or has blocked your crawler.