How to Build a Node.js Web Scraper

Step 1: Plan Your Scraper Architecture

Good architecture planning before writing code prevents expensive refactoring later. Three decisions shape your entire scraper design: what data you need, how the target site is structured, and which libraries to use.

Start by defining your data schema. List every field you want to extract and its data type. A product scraper might need: name (string), price (number), currency (string), URL (string), availability (boolean), rating (number), review count (number), and description (string). Writing this schema before you write any code forces you to think clearly about what you need and prevents scope creep during development.

Next, map the target site's structure. How are pages organized? Is there a listing page with links to individual product pages? How many pages of results are there? Does the listing page show all the data you need, or do you need to visit each product page individually? Is the content static or dynamic? Answering these questions determines your crawling strategy: whether you need one-level or two-level scraping, how many total requests you will make, and whether you need browser automation.

Finally, choose your library stack. For static pages, Axios and Cheerio provide the fastest and most resource-efficient approach. For dynamic pages that require JavaScript rendering, Playwright is the current best choice. For large-scale projects with thousands of URLs, Crawlee provides production infrastructure (request queuing, retries, proxy rotation) out of the box. If you are building from scratch rather than using Crawlee, plan for separate modules that handle fetching, parsing, storage, and orchestration. See our JavaScript scraping libraries comparison for detailed guidance on choosing the right tools.

Step 2: Set Up the Project

A well-organized project structure makes your scraper easier to develop, test, and maintain. Initialize your project with npm init -y, add "type": "module" to package.json for ES module syntax, and install your dependencies.

For a static scraper, install the core dependencies: npm install axios cheerio p-limit. Axios handles HTTP requests, Cheerio parses HTML, and p-limit controls concurrency. For utility needs, add npm install winston for structured logging and npm install json2csv if you need CSV output.

Organize your code into separate modules by responsibility. A fetcher.js module handles HTTP requests with headers, timeouts, retries, and proxy configuration. A parser.js module contains extraction functions that take HTML strings and return structured data objects. A storage.js module handles writing results to files or databases. A config.js module stores configuration like URLs, selectors, concurrency limits, and file paths. The main scraper.js file orchestrates these modules, managing the crawling loop, concurrency, and error handling.

This modular structure pays off immediately. When the target site changes its HTML, you update only parser.js. When you want to switch from JSON to database storage, you update only storage.js. When you need to add proxy rotation, you update only fetcher.js. No change requires touching the entire codebase.

Create a .env file for sensitive configuration like proxy credentials and API keys, and install dotenv to load environment variables. Add .env and node_modules to your .gitignore file to keep sensitive data and dependencies out of version control.

Step 3: Build the Fetching Layer

The fetching layer is responsible for downloading HTML pages reliably, handling errors gracefully, and respecting the target server's capacity. A well-built fetcher saves you from writing retry logic, header management, and proxy rotation in your main scraping code.

Start with an Axios instance configured with default settings. Create the instance with axios.create() and set a base timeout of 15 to 30 seconds, a default User-Agent header that mimics a real browser, and an Accept header with standard browser values. This instance becomes the foundation for all HTTP requests in your scraper.

Add request interceptors for dynamic header rotation. Instead of using the same User-Agent on every request, maintain an array of 10 to 20 realistic User-Agent strings from different browser versions and operating systems. In the request interceptor, randomly select one for each outgoing request. This simple step significantly reduces the likelihood of detection by bot protection systems that look for uniform request patterns.

Implement retry logic with exponential backoff. When a request fails with a transient error (5xx status codes, ECONNRESET, ETIMEDOUT), wait an increasing amount of time before retrying: 1 second after the first failure, 2 seconds after the second, 4 seconds after the third. Set a maximum of 3 to 5 retries. Log each retry attempt with the URL, error type, and retry number for debugging. Return null or throw a specific error for permanently failed URLs so the orchestration layer can handle them appropriately.

For proxy support, accept a proxy URL in the fetcher configuration and pass it to Axios via the proxy option or through an HTTPS agent. If you are using a rotating proxy service, each request automatically goes through a different IP address. If you are managing your own proxy pool, implement rotation logic that cycles through proxies and removes ones that return errors. For details on proxy strategies, see our guide on web scraping proxies.

Step 4: Build the Parsing Layer

The parsing layer converts raw HTML into structured data objects. Good parser design makes your selectors easy to update when the target site changes and keeps extraction logic separate from fetching and storage concerns.

Write a dedicated extraction function for each type of page you scrape. A product listing parser takes HTML and returns an array of objects, each with the fields visible on the listing page (name, price, URL). A product detail parser takes HTML and returns a single object with all the detailed fields (description, specifications, reviews). Keep each parser function focused on one page type.

Inside each parser function, load the HTML into Cheerio and use CSS selectors to locate elements. Build your data objects by extracting text with .text() and attributes with .attr(). Clean extracted values immediately: trim whitespace, parse numbers from price strings, resolve relative URLs to absolute ones, and normalize inconsistent formats. This cleaning step inside the parser means downstream code always receives consistent, usable data.

Add data validation to catch extraction errors early. After building a data object, check that required fields are not empty and that numeric fields contain valid numbers. If a field is missing or invalid, log a warning with the page URL and the problematic field. Depending on your requirements, you might skip the record, use a default value, or include it with a validation flag. Catching data quality issues at extraction time is far cheaper than discovering them after you have loaded thousands of records into a database.

Handle edge cases in the HTML. Some pages might have a different layout than others (featured products vs. regular products). Some elements might not exist on every page. Some text might contain HTML entities that need decoding. Test your parser against multiple real pages from the target site, not just one, to catch these variations before running the full scrape.

Step 5: Add Concurrency and Rate Limiting

Running requests one at a time is slow. Running them all at once will get you blocked or crash your process. Concurrency control lets you run multiple requests in parallel while staying within limits that the target server can handle and your system can sustain.

The p-limit library provides the simplest concurrency control for Node.js. Create a limiter with const limit = pLimit(5) to allow 5 concurrent requests. Wrap each fetch call with the limiter: limit(() => fetchPage(url)). The limiter automatically queues excess requests and runs them as slots become available. Use Promise.all() to wait for all queued requests to complete.

For more advanced control, p-queue provides a priority queue with configurable concurrency, interval-based rate limiting, and event hooks for monitoring progress. You can set both a concurrency limit (maximum simultaneous requests) and an interval limit (maximum requests per time period). For example, new PQueue({ concurrency: 3, interval: 1000, intervalCap: 3 }) runs at most 3 requests at a time with at most 3 requests per second.

Add delays between requests even with concurrency control. A constant delay (e.g., 500ms between each request) is the minimum. Randomized delays between 500ms and 2000ms look more like human behavior and are less likely to trigger pattern-based bot detection. Implement the delay as a simple function: const delay = (ms) => new Promise(r => setTimeout(r, ms)); and call it after each fetch completes.

Start with conservative concurrency settings (2 to 3 concurrent requests, 1 to 2 second delays) and increase gradually while monitoring for errors. If you start seeing 429 responses, CAPTCHA challenges, or connection timeouts, reduce your concurrency. Different websites have different thresholds, and there is no universal "right" setting. The goal is the highest throughput that the target server handles without pushing back.

Step 6: Implement Error Handling and Logging

Production scrapers run unattended, often overnight or on schedules. When something goes wrong, and it will, your logs are the only way to diagnose the problem without re-running the scraper. Invest in logging and error handling before you need them.

Use a structured logging library like Winston or Pino. Configure it to output JSON-formatted log entries with timestamps, log levels, and context fields. Log every request (URL, status code, response time), every extraction (page URL, items extracted), every error (URL, error type, stack trace), and every retry attempt. At the end of a run, log a summary: total pages processed, total items extracted, total errors, and elapsed time.

Categorize errors by recoverability. Transient errors (timeouts, 5xx responses, DNS failures) should be retried. Permanent errors (404, 403 after authentication) should be logged and skipped. Data quality errors (missing required fields, unparseable values) should be logged and the record should be flagged or excluded. This categorization prevents your scraper from wasting time retrying pages that will never succeed while still recovering from temporary network issues.

Implement graceful shutdown handling. When your scraper receives a SIGINT or SIGTERM signal (from Ctrl+C or a process manager), it should stop accepting new URLs, finish processing in-flight requests, save any buffered results, log a summary, and exit cleanly. In Node.js, register handlers with process.on('SIGINT', handler) and process.on('SIGTERM', handler). This prevents data loss when you need to stop a long-running scrape.

For long-running scrapes across thousands of pages, save results incrementally rather than holding everything in memory until the scrape completes. Write to a JSON Lines file (one JSON object per line) or insert into a database after every batch of pages. If the scraper crashes at page 5,000 of a 10,000-page scrape, you still have the first 5,000 results.

Step 7: Store and Export Results

Your storage strategy depends on the volume of data, how you will use the results downstream, and whether the scraper runs once or on an ongoing schedule.

For one-time or small-scale scrapes (up to a few thousand items), file-based storage is the simplest option. JSON files are the most natural choice for JavaScript scrapers. Use JSON Lines format (one JSON object per line, .jsonl extension) for incremental writes. Each line is a valid JSON object, so you can append new items without loading the entire file into memory. For CSV output, the json2csv package converts arrays of objects into CSV strings. Write the CSV file at the end of the scrape or stream rows as they are produced.

For recurring scrapes or larger datasets, database storage provides queryability, deduplication, and history tracking. SQLite (via the better-sqlite3 package) is an excellent single-file database for moderate volumes. Create a table matching your data schema, insert records as they are scraped, and use SQL queries to analyze results. For larger systems, PostgreSQL offers better concurrency, advanced indexing, and integration with analytics tools. The pg package or Prisma ORM provides clean Node.js interfaces for PostgreSQL.

Deduplication prevents the same item from being inserted multiple times across scraper runs. Define a unique key for each record (product URL, item ID, or a combination of fields) and check for existing records before inserting. In SQL databases, use UPSERT (INSERT ON CONFLICT UPDATE) to update existing records with new data. In file-based storage, maintain a set of previously seen keys and skip duplicates.

Data validation before storage catches issues early. Check that required fields are present and non-empty, that numeric fields parse correctly, that URLs are well-formed, and that text fields do not contain obvious garbage (like HTML tags or CSS class names that were accidentally included in the extraction). Log validation failures as warnings and decide whether to insert the record with a quality flag or skip it entirely.

Step 8: Deploy and Schedule

A scraper that only runs manually on your laptop is not a production system. Deployment and scheduling turn your scraper into an automated data pipeline that runs reliably without human intervention.

Containerize your scraper with Docker. Create a Dockerfile that starts from a Node.js base image, copies your project files, installs npm dependencies, and sets the entry point to your main scraper script. If you use Playwright or Puppeteer, use their official Docker images as base images, as these include pre-installed browser binaries. Docker ensures your scraper runs identically on your development machine, a CI server, and a production host.

For scheduling, the right tool depends on your infrastructure. On a dedicated server or VM, a cron job is the simplest option. Add an entry to your crontab that runs the scraper at your desired frequency. For more sophisticated scheduling with dependency management, retries, and a monitoring UI, use a job orchestration tool like Bull (Node.js-native), Apache Airflow, or Prefect.

Serverless platforms like AWS Lambda, Google Cloud Functions, and Azure Functions can run scrapers on a schedule without managing servers. The main constraint is execution time limits (15 minutes on Lambda). For short scrapes, serverless is the most cost-effective option because you pay only for the time your scraper actually runs. For longer scrapes, you need a persistent compute instance or a container service like AWS ECS or Google Cloud Run.

Set up monitoring and alerts. Track metrics like success rate, items scraped per run, average response time, and error count. Use a monitoring service (Datadog, Grafana Cloud, or even a simple Slack webhook) to send alerts when the error rate spikes or the item count drops below a threshold. These alerts let you respond to target site changes, blocking events, or infrastructure issues before they result in missing data.