How to Build a Web Crawler in Python

Step 1: Install Dependencies

You need two core libraries: requests for making HTTP calls and beautifulsoup4 for parsing HTML. Install them with pip:

pip install requests beautifulsoup4

The requests library provides a clean interface over Python's built-in urllib, handling sessions, cookies, redirects, and encoding automatically. BeautifulSoup wraps multiple HTML parsers (html.parser, lxml, html5lib) with a consistent API for navigating and searching the parse tree. For the parser backend, lxml is fastest but requires a C extension; the built-in html.parser works everywhere without additional dependencies.

Optionally install urllib3 for connection pooling (requests includes it) and tldextract for reliable domain extraction that handles edge cases like co.uk and com.au suffixes.

Step 2: Set Up the Crawler Structure

Create a Python class that holds the crawler's state: a deque for the URL frontier (pages waiting to be crawled), a set for tracking visited URLs (preventing duplicate fetches), and configuration parameters for user-agent string, crawl delay, and maximum page count.

from collections import deque
from urllib.parse import urlparse, urljoin, urldefrag
import time

class WebCrawler:
    def __init__(self, seeds, max_pages=100, delay=1.0):
        self.frontier = deque(seeds)
        self.visited = set()
        self.max_pages = max_pages
        self.delay = delay
        self.user_agent = "MyBot/1.0 (+https://example.com/bot)"
        self.results = []

The deque provides efficient O(1) append and popleft operations, making it ideal for a FIFO queue. The visited set uses Python's hash-based set for O(1) membership testing, critical when checking millions of URLs. For crawlers processing more than a few million URLs, consider a bloom filter which uses far less memory at the cost of a small false-positive rate.

Step 3: Implement robots.txt Parsing

Before crawling any domain, fetch and parse its robots.txt to determine which paths are allowed. Python's standard library includes urllib.robotparser which handles this:

from urllib.robotparser import RobotFileParser

def can_crawl(self, url):
    parsed = urlparse(url)
    robots_url = f"{parsed.scheme}://{parsed.netloc}/robots.txt"

    if robots_url not in self.robots_cache:
        rp = RobotFileParser()
        rp.set_url(robots_url)
        try:
            rp.read()
        except Exception:
            rp.allow_all = True
        self.robots_cache[robots_url] = rp

    return self.robots_cache[robots_url].can_fetch(
        self.user_agent, url
    )

Cache the parsed robots.txt per domain to avoid re-fetching it before every request. If the robots.txt cannot be fetched (404, timeout, or connection error), the standard practice is to assume all paths are allowed. Some crawlers take the conservative approach of blocking the entire domain when robots.txt is unreachable, but this would exclude many legitimate sites with misconfigured servers.

Step 4: Build the Fetch Function

The fetch function makes HTTP GET requests with proper headers, timeout handling, and error recovery. Set a descriptive User-Agent, configure reasonable timeouts, and handle common failure modes:

import requests

def fetch(self, url):
    headers = {"User-Agent": self.user_agent}
    try:
        response = requests.get(
            url,
            headers=headers,
            timeout=10,
            allow_redirects=True
        )
        if response.status_code == 200:
            content_type = response.headers.get("content-type", "")
            if "text/html" in content_type:
                return response.text
    except requests.RequestException:
        pass
    return None

The timeout parameter prevents the crawler from hanging indefinitely on unresponsive servers. Checking the Content-Type header ensures you only parse HTML pages, skipping PDFs, images, and other binary content that would fail HTML parsing. For production crawlers, add retry logic with exponential backoff for transient failures (network errors, 503 responses) and track response codes for monitoring.

Step 5: Extract and Normalize Links

Parse the fetched HTML with BeautifulSoup, find all anchor tags, extract their href attributes, and normalize each URL into a canonical absolute form:

from bs4 import BeautifulSoup

def extract_links(self, html, base_url):
    soup = BeautifulSoup(html, "html.parser")
    links = []
    for anchor in soup.find_all("a", href=True):
        href = anchor["href"]
        # Resolve relative URLs against the page URL
        absolute = urljoin(base_url, href)
        # Remove fragment identifiers
        defragged, _ = urldefrag(absolute)
        # Only keep http/https links
        if urlparse(defragged).scheme in ("http", "https"):
            links.append(defragged)
    return links

URL normalization is critical for deduplication. The urljoin function resolves relative paths (like "../page.html" or "/about") against the current page URL. The urldefrag function strips fragment identifiers (#section) since fragments reference positions within a page, not separate resources. Additional normalization steps for production crawlers include lowercasing the scheme and hostname, removing default port numbers (80 for HTTP, 443 for HTTPS), sorting query parameters alphabetically, and removing tracking parameters like utm_source.

Step 6: Add Politeness Controls

Implement per-domain rate limiting to avoid overwhelming any single server. Track the last request time for each domain and enforce a minimum delay between consecutive requests:

from urllib.parse import urlparse
import time

def wait_for_domain(self, url):
    domain = urlparse(url).netloc
    now = time.time()
    last_request = self.domain_timestamps.get(domain, 0)
    elapsed = now - last_request
    if elapsed < self.delay:
        time.sleep(self.delay - elapsed)
    self.domain_timestamps[domain] = time.time()

A one-second delay per domain is a conservative default that avoids triggering rate limits on most websites. Some robots.txt files specify a crawl-delay directive indicating the site's preferred interval. For crawlers that visit many different domains, the per-domain delay has minimal impact on overall throughput since the crawler can process requests to other domains while waiting. For single-domain crawls, the delay directly limits throughput, so finding the right balance between speed and politeness matters.

Step 7: Store Crawled Data

Save page content and metadata in a structured format. For small crawls, JSON lines (one JSON object per line) is simple and effective. For larger crawls, SQLite provides queryable storage without requiring a database server:

import json

def store_page(self, url, html):
    soup = BeautifulSoup(html, "html.parser")
    title = soup.title.string if soup.title else ""
    meta_desc = ""
    meta_tag = soup.find("meta", attrs={"name": "description"})
    if meta_tag:
        meta_desc = meta_tag.get("content", "")

    record = {
        "url": url,
        "title": title.strip(),
        "description": meta_desc,
        "word_count": len(soup.get_text().split()),
        "crawled_at": time.time()
    }
    self.results.append(record)

Storing both metadata and raw HTML gives you flexibility. The metadata enables quick queries and analysis without reparsing, while the raw HTML allows you to extract additional information later as your needs evolve. For disk-based storage, consider compressing HTML with gzip since web pages compress extremely well (typically 70-90% size reduction).

Step 8: Run the Crawl Loop

Connect all components into the main crawl loop. The loop pulls URLs from the frontier, checks robots.txt permissions, waits for politeness delays, fetches the page, extracts and queues new links, and stores results:

def crawl(self):
    while self.frontier and len(self.visited) < self.max_pages:
        url = self.frontier.popleft()
        if url in self.visited:
            continue
        if not self.can_crawl(url):
            continue

        self.wait_for_domain(url)
        html = self.fetch(url)
        if html is None:
            continue

        self.visited.add(url)
        self.store_page(url, html)

        for link in self.extract_links(html, url):
            if link not in self.visited:
                self.frontier.append(link)

    return self.results

This loop implements the fundamental crawl cycle that every crawler uses, from simple scripts to distributed systems processing billions of pages. The order of checks matters: skip visited URLs first (cheapest check), then robots.txt (cached lookup), then wait and fetch (network I/O). For production use, add logging at each stage so you can diagnose issues like unexpected blocks, high error rates, or frontier exhaustion.