Scrapy Tutorial for Beginners

Prerequisites

Before starting with Scrapy, you need Python 3.8 or later installed on your system. You should be comfortable with basic Python concepts like classes, functions, dictionaries, and importing modules. You do not need prior web scraping experience, though a basic understanding of HTML structure (tags, attributes, nesting) will help you write selectors more confidently.

Scrapy runs on Windows, macOS, and Linux. The examples in this tutorial use terminal commands that work on macOS and Linux. Windows users can follow along using PowerShell or Windows Subsystem for Linux (WSL), which provides a more consistent experience with Scrapy's tooling.

Installing Scrapy

Always install Scrapy inside a virtual environment to avoid dependency conflicts with other Python projects. Create and activate a virtual environment first:

python -m venv scraping-env
source scraping-env/bin/activate   # macOS/Linux
scraping-env\Scripts\activate      # Windows

Then install Scrapy with pip:

pip install scrapy

The installation pulls in several dependencies including lxml (for HTML parsing), Twisted (for async networking), and parsel (for CSS/XPath selectors). If installation fails on Linux, you may need system development packages:

sudo apt-get install python3-dev libxml2-dev libxslt1-dev libffi-dev libssl-dev

Verify the installation by checking the version:

scrapy version

You should see output like "Scrapy 2.16.0" or similar. If this works, your environment is ready for your first project.

Creating a Scrapy Project

Scrapy organizes code into projects with a standard directory structure. Create one using the startproject command:

scrapy startproject tutorial
cd tutorial

This generates the following structure:

tutorial/
    scrapy.cfg            # deployment configuration
    tutorial/
        __init__.py
        items.py          # data model definitions
        middlewares.py    # request/response processors
        pipelines.py     # item processing stages
        settings.py      # project configuration
        spiders/          # your spider classes go here
            __init__.py

Each file serves a specific purpose. For this beginner tutorial, you will primarily work with the spiders/ directory where your crawler logic lives, and settings.py for basic configuration. The other files become important as your projects grow more complex.

Writing Your First Spider

A spider is a Python class that tells Scrapy where to start crawling, how to follow links, and what data to extract from each page. Create a new file called quotes_spider.py in the spiders/ directory:

import scrapy

class QuotesSpider(scrapy.Spider):
    name = "quotes"
    start_urls = [
        "https://quotes.toscrape.com/page/1/",
    ]

    def parse(self, response):
        for quote in response.css("div.quote"):
            yield {
                "text": quote.css("span.text::text").get(),
                "author": quote.css("small.author::text").get(),
                "tags": quote.css("div.tags a.tag::text").getall(),
            }

        next_page = response.css("li.next a::attr(href)").get()
        if next_page is not None:
            yield response.follow(next_page, callback=self.parse)

Let us break down each part of this spider:

name: A unique identifier for this spider. You use this name when running the spider from the command line. Each spider in a project must have a different name.

start_urls: A list of URLs where Scrapy begins crawling. The framework downloads these pages first and passes the responses to the parse method.

parse method: This is the callback that processes each downloaded page. It receives a response object containing the page HTML, headers, and URL. Your job is to extract data from this response.

yield dictionary: When you yield a dictionary, Scrapy treats it as a scraped item. The item flows through your configured pipelines and eventually reaches the output (file, database, or other destination).

response.follow: This generates a new request for Scrapy to download. The callback parameter tells Scrapy which method should process the response. Here we reuse self.parse to handle pagination pages identically to the first page.

Understanding CSS Selectors

CSS selectors are the primary way to locate elements on a page. Scrapy extends standard CSS selectors with pseudo-elements for extracting text and attributes. The most common patterns you will use as a beginner are:

# Select element by class
response.css("div.quote")

# Get text content of an element
response.css("span.text::text").get()

# Get an attribute value
response.css("a::attr(href)").get()

# Get all matches as a list
response.css("a.tag::text").getall()

# Chain selectors (select within a selection)
quote = response.css("div.quote")[0]
quote.css("span.text::text").get()

The ::text pseudo-element extracts the text content of an element. Without it, you get the element itself rather than its text. The ::attr(name) pseudo-element extracts the value of a specified attribute, commonly used for href links and src image paths.

The .get() method returns the first match or None if nothing matches. The .getall() method returns a list of all matches, which can be empty. Use .get() when you expect a single value and .getall() when there could be multiple results.

To test selectors before writing them into your spider, use the Scrapy shell:

scrapy shell "https://quotes.toscrape.com"

This downloads the page and opens an interactive Python session where you can experiment with selectors against the live response. It saves significant time compared to running your full spider to test each selector.

Running Your Spider

Run the spider from your project root directory:

scrapy crawl quotes

This starts the crawl and prints scraped items to the terminal along with log messages showing Scrapy's activity: requests sent, responses received, items scraped, and any errors encountered.

To save output to a file, use the -O flag with a filename:

scrapy crawl quotes -O quotes.json

Scrapy determines the output format from the file extension. Supported formats include:

scrapy crawl quotes -O quotes.json       # JSON array
scrapy crawl quotes -O quotes.jl         # JSON Lines (one object per line)
scrapy crawl quotes -O quotes.csv        # Comma-separated values
scrapy crawl quotes -O quotes.xml        # XML format

JSON Lines (.jl) is recommended for large crawls because it can be appended to without loading the entire file, and it is easy to process line by line with standard tools.

Configuring Basic Settings

The settings.py file controls how Scrapy behaves during a crawl. For beginners, the most important settings are:

# Respect robots.txt rules
ROBOTSTXT_OBEY = True

# Add a delay between requests to the same site
DOWNLOAD_DELAY = 1

# Limit concurrent requests
CONCURRENT_REQUESTS = 8

# Identify your spider in request headers
USER_AGENT = "tutorial-spider (+http://www.yourdomain.com)"

ROBOTSTXT_OBEY tells Scrapy to check each site's robots.txt before crawling and respect any disallow rules. This is enabled by default and should remain on for ethical scraping.

DOWNLOAD_DELAY sets the minimum time in seconds between consecutive requests to the same domain. Setting this to 1 or higher prevents overwhelming target servers and reduces the chance of getting blocked.

CONCURRENT_REQUESTS limits how many simultaneous downloads Scrapy performs. Lower values are gentler on target servers. Higher values speed up crawling but increase resource usage and block risk.

Handling Common Beginner Issues

Several issues frequently trip up beginners working with Scrapy for the first time.

Empty results: If your spider yields nothing, the most likely cause is incorrect selectors. Use scrapy shell to verify your CSS or XPath expressions match the actual page HTML. Remember that some websites load content via JavaScript, which Scrapy does not render by default.

403 Forbidden errors: Many websites block requests that lack a proper User-Agent header or that come too quickly. Set a realistic USER_AGENT in settings and add a DOWNLOAD_DELAY of at least 1 second.

Duplicate requests ignored: Scrapy automatically filters duplicate URLs within a crawl. If you see log messages about filtered duplicates, your link-following logic may be yielding the same URLs repeatedly. This is normal behavior and usually not a problem.

Encoding issues: Some pages use non-UTF-8 encodings. Scrapy usually detects encoding automatically, but if you see garbled text in your output, you can force encoding with response.css("selector::text").get().encode("latin-1").decode("utf-8") or similar transformations in your parse method.

Next Steps After This Tutorial

Once you are comfortable with basic spiders, you can expand your skills in several directions. Learn about item pipelines to process and store your data in databases. Explore advanced spider patterns including CrawlSpider for rule-based crawling. Understand how proxy integration helps you scale to larger crawls without getting blocked.

The Scrapy documentation at docs.scrapy.org provides comprehensive reference material. The official tutorial covers similar ground to this page but with additional detail on some advanced topics. Community resources on GitHub and Stack Overflow address most issues you will encounter as your projects grow.