Scrapy vs Playwright for Scraping

How Each Tool Approaches Scraping

Scrapy sends HTTP requests directly to web servers and receives HTML responses. It never launches a browser, never executes JavaScript, and never renders CSS. When you scrape a page with Scrapy, you get the same raw HTML that a search engine crawler would receive. This approach is efficient because it skips the expensive rendering step, but it cannot access content that requires JavaScript execution to appear in the DOM.

Playwright controls a real browser instance (Chromium, Firefox, or WebKit) programmatically. When you navigate to a page with Playwright, the browser downloads all resources, executes all JavaScript, renders the page visually, and produces a fully-constructed DOM identical to what a human user sees. This means Playwright can scrape single-page applications, infinite scroll feeds, content loaded via AJAX calls, and pages behind JavaScript-based authentication flows.

The fundamental tradeoff is speed and resources versus page compatibility. Scrapy processes thousands of pages per minute using minimal memory and CPU. Playwright processes tens of pages per minute while consuming significant memory (each browser tab uses 50-200MB of RAM) and CPU for rendering. However, Playwright can access content that Scrapy simply cannot reach without browser execution.

Performance and Resource Usage

The performance gap between these tools is substantial and directly impacts infrastructure costs and crawl completion times.

A single Scrapy process can comfortably handle 200-500 concurrent requests on modest hardware, downloading and parsing thousands of pages per minute. Memory usage stays relatively flat regardless of crawl size because Scrapy processes items through pipelines and discards them. A complete e-commerce crawl of 100,000 product pages might take 30 minutes with Scrapy running at moderate concurrency with politeness delays.

Playwright running the same 100,000 pages would require dramatically more resources. Each browser context consumes 50-200MB of RAM, limiting practical concurrency to perhaps 10-20 simultaneous pages on a machine with 16GB of memory. With JavaScript execution, page rendering, and resource loading, each page takes 2-5 seconds compared to Scrapy's 200-500 milliseconds. The same crawl might take 10-20 hours with Playwright, even with aggressive parallelization.

Network bandwidth tells a similar story. Scrapy downloads only the HTML document (typically 50-200KB per page). Playwright downloads everything the browser needs: HTML, CSS, JavaScript bundles, images, fonts, analytics scripts, and third-party resources. A page that costs 100KB in Scrapy might consume 2-5MB in Playwright due to all associated resources being loaded.

JavaScript Rendering Capabilities

The defining advantage of Playwright is its ability to handle JavaScript-dependent content that Scrapy misses entirely.

Single-page applications (SPAs): React, Vue, and Angular applications render their content client-side. The initial HTML response contains an empty container element and a JavaScript bundle. Without executing that JavaScript, there is no content to extract. Playwright renders these applications fully, exposing all dynamically generated DOM elements for scraping.

Infinite scroll and lazy loading: Content that loads as the user scrolls requires scroll interactions and waiting for AJAX responses. Playwright can programmatically scroll the page, wait for network requests to complete, and extract the newly loaded content. Scrapy sees only what appears in the initial HTML response.

Interactive elements: Pages that require clicking buttons, filling forms, selecting dropdowns, or hovering over elements to reveal content need browser interaction. Playwright provides a full interaction API for these actions. Scrapy cannot interact with page elements since it never renders them.

Authentication flows: JavaScript-based login systems, OAuth redirects, and CAPTCHA challenges require a browser environment. Playwright handles these by navigating through the login process like a human user, then scraping authenticated pages. Scrapy can handle simple cookie-based authentication but struggles with JavaScript-dependent auth flows.

Anti-Bot Detection

Modern anti-bot systems differ in how they detect Scrapy versus Playwright, and neither tool is invisible by default.

Scrapy is detectable because it does not execute JavaScript. Anti-bot systems that serve JavaScript challenges (like Cloudflare's browser verification) will block Scrapy requests immediately since there is no browser to solve the challenge. However, Scrapy's HTTP-level requests are simple and leave minimal fingerprint beyond headers and IP address.

Playwright runs a real browser but automated browsers have detectable characteristics. Default Playwright instances expose automation flags (navigator.webdriver = true), use non-standard viewport sizes, lack browser extensions, and exhibit timing patterns that differ from human behavior. Anti-bot systems from DataDome, PerimeterX, and Kasada specifically look for these automation signals.

Stealth techniques exist for both tools. Scrapy benefits from proper header sets, proxy rotation, and realistic request timing. Playwright benefits from stealth plugins that patch detectable browser properties, randomized viewport sizes, and human-like interaction delays. Neither tool is inherently better at evasion, they just face different detection vectors.

Project Architecture and Maintenance

Scrapy provides a full project framework with standardized locations for spiders, pipelines, middleware, and settings. Multiple developers can work on a Scrapy project with clear conventions. Adding new spiders, modifying data processing, or changing storage destinations follows established patterns. The middleware system handles cross-cutting concerns like proxy rotation and retry logic without modifying spider code.

Playwright scripts are typically standalone Python files without a prescribed project structure. For small scraping tasks this is fine, but as projects grow you build your own framework around Playwright for scheduling, error handling, data processing, and storage. The playwright-python library provides browser control but not the scraping infrastructure that Scrapy includes.

Maintenance differs as well. Scrapy spiders break when HTML structure changes, requiring selector updates. Playwright scripts break for the same reason plus browser version changes, rendering timing issues, and JavaScript framework updates that alter DOM construction timing. Playwright scripts tend to be more brittle because they depend on exact page rendering behavior rather than just HTML structure.

When to Choose Scrapy Alone

Scrapy is the clear winner for projects where target content exists in the initial HTML response:

Server-rendered websites: Traditional websites built with PHP, Ruby on Rails, Django, or any server-side framework deliver complete HTML without JavaScript. Blogs, news sites, government databases, academic repositories, and many e-commerce platforms serve content this way.

High-volume crawls: Any project requiring thousands or millions of pages needs Scrapy's efficiency. The resource overhead of browser automation makes Playwright impractical at this scale without significant infrastructure investment.

API scraping: When the target data comes from REST APIs rather than rendered pages, Scrapy's HTTP client handles JSON responses efficiently without any need for browser rendering.

Structured data feeds: Sitemaps, RSS feeds, XML catalogs, and other machine-readable formats are perfectly suited to Scrapy's HTTP-level approach.

When to Choose Playwright

Playwright is necessary when the target content requires browser execution to exist:

SPAs and JavaScript-heavy sites: Applications built entirely in React, Vue, or Angular that render content client-side require a browser to produce scrapeable DOM.

Interactive content: Pages where data appears only after user interactions (clicks, scrolls, form submissions) need Playwright's interaction capabilities.

Anti-bot JavaScript challenges: Sites protected by Cloudflare, DataDome, or similar services that require JavaScript execution to access content need a real browser.

Small-scale precision scraping: When you need data from a handful of complex pages rather than thousands of simple ones, Playwright's overhead is acceptable and its rendering guarantees prevent missed content.

Combining Scrapy and Playwright

The most powerful approach for many projects combines both tools through the scrapy-playwright integration. This plugin adds Playwright rendering to specific requests within a Scrapy crawl while leaving simple pages to Scrapy's efficient HTTP client.

import scrapy

class HybridSpider(scrapy.Spider):
    name = "hybrid"

    def start_requests(self):
        # Use Playwright only for pages that need it
        yield scrapy.Request(
            "https://spa-site.com/products",
            meta={"playwright": True, "playwright_include_page": True},
            callback=self.parse_spa,
        )

        # Use regular Scrapy for static pages
        yield scrapy.Request(
            "https://static-site.com/catalog",
            callback=self.parse_static,
        )

    async def parse_spa(self, response):
        page = response.meta["playwright_page"]
        await page.wait_for_selector("div.product-grid")
        # Now extract from the fully-rendered page
        for product in response.css("div.product-card"):
            yield {"name": product.css("h2::text").get()}
        await page.close()

    def parse_static(self, response):
        for product in response.css("div.product-card"):
            yield {"name": product.css("h2::text").get()}

This hybrid approach gives you Scrapy's scheduling, pipelines, and middleware infrastructure while using Playwright's rendering only where needed. Pages that work without JavaScript use Scrapy's fast HTTP client. Pages that require browser execution get Playwright rendering. You configure this per-request, not globally, so you optimize resource usage across the entire crawl.