Crawlee Web Scraping Framework Guide

What Crawlee Does

Crawlee solves the infrastructure problem in web scraping. When you build a scraper from scratch with Axios and Cheerio or with Puppeteer, you need to write your own code for managing URL queues, retrying failed requests, rotating proxies, handling concurrency, storing results, and recovering from crashes. Crawlee provides all of this out of the box, letting you focus on the extraction logic rather than the plumbing.

The framework is the JavaScript equivalent of Python's Scrapy, but with a significant advantage: Crawlee supports both static HTML scraping (via Cheerio) and browser-based scraping (via Puppeteer or Playwright) through a unified API. You can switch between scraping modes by changing the crawler class you instantiate, without rewriting your extraction code. This flexibility makes Crawlee uniquely suited for projects that need to scrape a mix of static and dynamic websites.

Crawlee was originally known as Apify SDK and was tightly coupled to the Apify cloud platform. The rebranding and open-source release in 2022 separated the core framework from the cloud platform, making Crawlee usable as a standalone tool that runs anywhere: on your laptop, on a VPS, in a Docker container, or on Apify's managed infrastructure. The framework has seen steady adoption growth, with over 15,000 GitHub stars as of mid-2026.

The Three Crawler Types

Crawlee provides three crawler classes that share the same API for request management, data storage, and error handling. The difference is how they fetch and render pages.

CheerioCrawler

CheerioCrawler fetches pages using an HTTP client (similar to Axios) and parses them with Cheerio. It is the fastest option because it does not launch a browser. A single CheerioCrawler instance can process 50 to 100 pages per second on a typical server, making it suitable for scraping millions of static pages in a reasonable timeframe.

CheerioCrawler is the right choice when the target website serves its content as static HTML. It uses significantly less memory and CPU than the browser-based crawlers, typically running with 100 to 200 MB of RAM compared to 500 MB to 2 GB for browser crawlers. Use CheerioCrawler for blogs, news sites, documentation, directories, traditional e-commerce listings, and any page where the data is present in the initial HTML response.

Inside the request handler, Crawlee provides a Cheerio $ object already loaded with the page's HTML. You use the same CSS selectors and extraction methods described in our Cheerio guide. The handler also receives the request URL, response headers, and access to Crawlee's data storage and request queue for adding new URLs to crawl.

PlaywrightCrawler

PlaywrightCrawler launches browser instances using Playwright and renders pages with full JavaScript execution. It supports Chromium, Firefox, and WebKit, giving you cross-browser scraping capability. Each page gets its own browser context, which provides isolation between requests and prevents cookie or session leakage across different scraping tasks.

PlaywrightCrawler is the recommended choice for dynamic websites that rely on JavaScript to render content. It handles single-page applications, infinite scroll, AJAX-loaded data, and interactive elements automatically. Playwright's auto-waiting ensures that elements are visible and stable before your extraction code runs, reducing flaky behavior compared to manual waiting strategies.

The request handler receives a Playwright page object with full access to the Playwright API. You can call page.evaluate() to extract data from the rendered DOM, page.click() to interact with elements, and page.waitForSelector() for explicit waiting. Crawlee handles browser lifecycle management (launching, closing, restarting crashed browsers) automatically.

PuppeteerCrawler

PuppeteerCrawler is functionally similar to PlaywrightCrawler but uses Google's Puppeteer library instead of Playwright. It supports only Chrome and Chromium. PuppeteerCrawler exists primarily for backward compatibility and for teams that have existing Puppeteer code. For new projects, PlaywrightCrawler is the recommended choice due to Playwright's better auto-waiting, cross-browser support, and more active development.

Request Queue and URL Management

Crawlee's request queue is one of its most valuable features. It automatically handles URL deduplication, prioritization, retry logic, and persistence across process restarts.

You add URLs to the queue using crawler.addRequests() or by calling context.enqueueLinks() inside a request handler. The enqueueLinks() method is particularly powerful: it finds all links on the current page matching a specified pattern and adds them to the queue automatically. You can filter links by URL pattern (glob or regex), limit the crawl to the same domain, or exclude specific URL patterns. This makes it trivial to build crawlers that discover and follow links across an entire website.

URL deduplication is automatic. If you add the same URL twice, Crawlee recognizes the duplicate and processes it only once. It normalizes URLs before comparison, handling differences in trailing slashes, query parameter order, and URL encoding. This prevents wasted requests when multiple pages link to the same destination.

The queue persists to disk by default. If your crawler crashes after processing 50,000 URLs, you can restart it and it picks up where it left off without re-processing already-completed URLs. This persistence is critical for large crawls that take hours to complete, as it eliminates the need for external checkpoint management.

Request priority lets you process certain URLs before others. You can assign numeric priorities when adding requests, and the queue processes higher-priority requests first. This is useful for two-level scraping patterns where you want to process listing pages (which generate detail page URLs) before detail pages, or for prioritizing high-value pages in a crawl.

Proxy Rotation and Session Management

Crawlee includes built-in support for proxy rotation and session management, which are essential for scraping at scale without getting blocked.

Proxy configuration accepts a list of proxy URLs and automatically rotates through them. You can provide datacenter proxies, residential proxies, or a mix of both. Crawlee tracks which proxies are working and which are returning errors, automatically retiring unhealthy proxies and redistributing requests to healthy ones. For Apify's proxy service, Crawlee includes native integration that handles authentication and rotation automatically.

The session management system ties together proxies, cookies, and browser fingerprints into coherent "sessions" that represent a single virtual user. Each session maintains its own cookie jar, uses a consistent proxy, and (for browser crawlers) presents a consistent fingerprint. When a session starts receiving errors (blocked responses, CAPTCHAs, rate limits), Crawlee retires the entire session and creates a new one with a fresh proxy and clean cookies. This session rotation makes it much harder for target sites to identify and block your scraper.

The SessionPool manages a configurable number of concurrent sessions, creating new ones as needed and retiring ones that fail. You configure the maximum number of sessions, the maximum age of a session before automatic retirement, and the error threshold that triggers early retirement. This system runs automatically once configured, requiring no per-request management from your code.

Data Storage

Crawlee provides a Dataset class for storing scraped results. You push items to the dataset from inside your request handler using context.pushData(item), and Crawlee handles storage, serialization, and export.

By default, datasets are stored as JSON files on disk. Each item is persisted immediately when pushed, which means no data is lost if the crawler crashes. After the crawl completes, you can export the entire dataset to JSON, CSV, or other formats using Crawlee's built-in export functions.

Crawlee also provides a KeyValueStore for storing arbitrary key-value data like configuration, intermediate state, or page screenshots. The key-value store is useful for saving metadata about the crawl (start time, configuration, error summary) alongside the scraped data.

When running on the Apify platform, datasets and key-value stores are automatically backed up to cloud storage, accessible through the Apify API and web UI. When running locally, they are stored in a ./storage directory in your project folder.

Error Handling and Retries

Crawlee's error handling is automatic and configurable. When a request handler throws an error (due to a network failure, a parsing error, or any other exception), Crawlee catches it, logs the error, and adds the request back to the queue for retry. Failed requests are retried up to a configurable maximum (default is 3 retries), with increasing delays between attempts.

After exhausting all retries, the request is moved to a "failed requests" list that you can inspect after the crawl completes. This lets you identify pages that consistently fail and investigate the cause, whether it is a permanent block, a page that no longer exists, or a parsing bug in your handler.

For browser-based crawlers, Crawlee also handles browser crashes gracefully. If a browser instance crashes or becomes unresponsive, Crawlee kills the process, launches a new browser, and retries the request. This resilience is important for long-running crawls where browser memory leaks or rendering hangs can occur after processing thousands of pages.

You can customize retry behavior per request by catching specific errors in your handler and calling context.request.noRetry = true for errors that should not be retried (like 404 pages or access denied responses). This prevents wasting time retrying URLs that will never succeed.

Getting Started with Crawlee

Crawlee provides a CLI tool for scaffolding new projects. Run npx crawlee create my-scraper to generate a project with all the boilerplate already configured. The CLI offers templates for each crawler type, so you can start with a CheerioCrawler, PlaywrightCrawler, or PuppeteerCrawler template depending on your needs.

The core of a Crawlee scraper is the request handler function. This function receives a context object with the parsed page (a Cheerio $ object or a Playwright page), the request URL and metadata, and helper methods for adding URLs to the queue and pushing data to the dataset. Your job is to write the extraction logic inside this handler. Everything else, fetching, retrying, queuing, proxy rotation, concurrency, and data storage, is handled by the framework.

A minimal CheerioCrawler scraper creates a new CheerioCrawler instance with a requestHandler function, adds starting URLs with crawler.addRequests(), and calls crawler.run(). Inside the handler, you use $ (Cheerio) to extract data and context.pushData() to save results. The framework handles everything else. Adding proxy rotation, session management, or concurrency tuning requires only configuration changes, not code changes.

When to Use Crawlee vs Building from Scratch

Crawlee adds value for any project that involves more than a handful of pages or that needs to run reliably without manual oversight. Its request queue, retry logic, and crash recovery eliminate the most common production scraping issues that custom-built scrapers struggle with.

Building from scratch with Axios and Cheerio makes sense for simple, one-off scraping tasks where you need to fetch 10 to 50 pages and extract data quickly. The setup time is near zero, and the simpler architecture is easier to understand and debug for small projects.

The break-even point is roughly 100 to 500 pages. Below that, the overhead of setting up Crawlee is not worth it. Above that, the infrastructure Crawlee provides (especially the persistent request queue and automatic retries) saves more development time than the initial setup costs. For thousands of pages or recurring scrapes, Crawlee is almost always the better choice.

Crawlee also makes sense when you are uncertain whether your targets are static or dynamic. You can start with CheerioCrawler for speed, and if you discover that some pages need JavaScript rendering, switch to PlaywrightCrawler by changing one class name. Your request handler code, URL management, and data storage remain unchanged.