How to Build Scrapy Spiders

Define the Spider Class

Every spider starts with a class that inherits from scrapy.Spider. The class needs three essential elements: a unique name string used to run the spider from the command line, a start_urls list containing one or more URLs where crawling begins, and a parse method that processes the initial responses.

import scrapy

class ProductSpider(scrapy.Spider):
    name = "products"
    start_urls = [
        "https://shop.example.com/category/electronics",
        "https://shop.example.com/category/books",
    ]

    def parse(self, response):
        # Extraction logic goes here
        pass

The name must be unique within your project. Scrapy uses it to locate and run the spider: scrapy crawl products. The start_urls can contain multiple seed URLs if your crawl should begin from several starting points. Scrapy downloads all start URLs and passes each response to the parse method independently.

For more control over initial requests, override the start_requests method instead of defining start_urls. This lets you set custom headers, cookies, or metadata on the initial requests:

def start_requests(self):
    urls = ["https://shop.example.com/api/products?page=1"]
    for url in urls:
        yield scrapy.Request(
            url,
            headers={"Accept": "application/json"},
            callback=self.parse,
        )

Write Extraction Logic

Inside your parse method, use CSS or XPath selectors to locate and extract data from the HTML response. Scrapy's selector system is built on lxml and parsel, providing fast, reliable parsing with a clean API.

def parse(self, response):
    for product in response.css("div.product-card"):
        yield {
            "name": product.css("h2.title::text").get(),
            "price": product.css("span.price::text").get(),
            "url": product.css("a::attr(href)").get(),
            "rating": product.css("div.stars::attr(data-rating)").get(),
            "in_stock": product.css("span.stock::text").get() is not None,
        }

Each yield of a dictionary creates an item that flows through your configured pipelines. Use .get() for single values and .getall() for lists. Always handle potentially missing elements by checking for None or providing defaults:

# Safe extraction with default values
price = product.css("span.price::text").get(default="0.00")
tags = product.css("a.tag::text").getall() or ["uncategorized"]

For complex pages where CSS selectors are insufficient, switch to XPath for its more powerful navigation capabilities. XPath can traverse up the DOM tree, select by text content, and handle sibling relationships that CSS cannot express.

Implement Link Following

Most scraping projects need to follow links to pagination pages, detail pages, or subcategories. Yield new requests from your parse method using response.follow() for relative URLs or scrapy.Request() for absolute URLs.

def parse(self, response):
    # Extract items from the listing page
    for product in response.css("div.product-card"):
        detail_url = product.css("a::attr(href)").get()
        yield response.follow(detail_url, callback=self.parse_detail)

    # Follow pagination
    next_page = response.css("a.next-page::attr(href)").get()
    if next_page:
        yield response.follow(next_page, callback=self.parse)

The response.follow() method resolves relative URLs automatically against the current page URL. It also handles common URL patterns like relative paths, protocol-relative URLs, and fragment identifiers. For pagination, recursively calling the same parse method creates a loop that continues until no more "next page" links are found.

You can pass data between pages using the meta parameter. This is useful when listing pages contain data (like category name) that you want attached to items extracted from detail pages:

yield response.follow(
    detail_url,
    callback=self.parse_detail,
    meta={"category": "electronics", "listing_price": price},
)

Add Multiple Callbacks

Real-world spiders typically handle several page types, each with different HTML structures and data to extract. Create separate methods for each page type and route responses to the appropriate callback.

class EcommerceSpider(scrapy.Spider):
    name = "ecommerce"
    start_urls = ["https://shop.example.com/"]

    def parse(self, response):
        """Handle category listing pages."""
        for link in response.css("a.category-link::attr(href)").getall():
            yield response.follow(link, callback=self.parse_category)

    def parse_category(self, response):
        """Handle product listing within a category."""
        for product in response.css("div.product-card"):
            link = product.css("a::attr(href)").get()
            yield response.follow(link, callback=self.parse_product)

        # Pagination within category
        next_page = response.css("a.next::attr(href)").get()
        if next_page:
            yield response.follow(next_page, callback=self.parse_category)

    def parse_product(self, response):
        """Extract full product details."""
        yield {
            "name": response.css("h1.product-name::text").get(),
            "price": response.css("span.final-price::text").get(),
            "description": " ".join(response.css("div.description p::text").getall()),
            "sku": response.css("span.sku::text").get(),
            "images": response.css("img.product-image::attr(src)").getall(),
            "url": response.url,
            "category": response.meta.get("category", "unknown"),
        }

This three-level pattern (homepage to categories, categories to products, products to data) is the most common architecture for e-commerce and directory scraping. Each callback is responsible for one page type, making the code easy to debug and maintain. If a category page changes layout, you only modify parse_category without touching the product extraction logic.

Handle Errors and Edge Cases

Production spiders encounter network timeouts, missing pages, changed layouts, and blocked requests. Add error handling to make your spiders resilient to these failures without crashing.

def start_requests(self):
    for url in self.start_urls:
        yield scrapy.Request(
            url,
            callback=self.parse,
            errback=self.handle_error,
        )

def handle_error(self, failure):
    self.logger.error(f"Request failed: {failure.request.url}")
    self.logger.error(f"Reason: {failure.value}")

def parse_product(self, response):
    # Check for soft 404s or blocked pages
    if response.css("div.captcha-challenge"):
        self.logger.warning(f"Captcha detected on {response.url}")
        return

    name = response.css("h1::text").get()
    if not name:
        self.logger.warning(f"No product name found on {response.url}")
        return

    yield {
        "name": name.strip(),
        "price": response.css("span.price::text").get(default="N/A"),
        "url": response.url,
    }

The errback parameter on requests catches network-level failures (timeouts, DNS errors, connection refused) that never produce a response. The callback handles HTTP-level issues by inspecting the response content for signs of blocking or unexpected page structures.

Always validate extracted data before yielding items. Check that critical fields are present and non-empty. Log warnings for unexpected page states so you can diagnose issues during crawl review. Scrapy's built-in logging integrates with Python's logging module, so you can direct spider logs to files, monitoring systems, or alerting pipelines.

Use CrawlSpider for Rule-Based Crawling

When your crawl logic follows consistent URL patterns, CrawlSpider provides a declarative approach using Rule objects and link extractors. Instead of manually yielding requests in each callback, you define rules that automatically extract and follow links matching specified patterns.

from scrapy.spiders import CrawlSpider, Rule
from scrapy.linkextractors import LinkExtractor

class AutoCrawler(CrawlSpider):
    name = "autocrawl"
    start_urls = ["https://shop.example.com/"]
    allowed_domains = ["shop.example.com"]

    rules = (
        # Follow category links, do not extract data from them
        Rule(LinkExtractor(allow=r"/category/\w+")),
        # Follow product links and extract data
        Rule(
            LinkExtractor(allow=r"/product/\d+"),
            callback="parse_product",
        ),
    )

    def parse_product(self, response):
        yield {
            "name": response.css("h1::text").get(),
            "price": response.css("span.price::text").get(),
            "url": response.url,
        }

CrawlSpider automatically follows links matching your rules without explicit request yielding. Rules without a callback just follow links (for navigation pages). Rules with a callback extract data from matching pages. The allowed_domains setting prevents the spider from following external links off the target site.

LinkExtractor accepts allow and deny parameters with regular expressions, plus restrict_css to limit extraction to specific page regions. This combination handles most crawling patterns without writing procedural link-following code.