JavaScript Web Scraping Tutorial

Step 1: Set Up Your Node.js Project

Before writing any scraping code, you need Node.js installed on your system. Open a terminal and run node --version to check. You need Node.js 18 or later, as older versions lack the built-in fetch API and have reached end-of-life for security updates. If Node.js is not installed, download it from the official nodejs.org website. The LTS (Long Term Support) version is the recommended choice for most users.

Create a new directory for your project and navigate into it. Run npm init -y to generate a package.json file with default settings. This file tracks your project's metadata and dependencies. Open the generated package.json and add "type": "module" to enable ES module syntax (import/export) instead of the older CommonJS require() syntax. ES modules are the modern standard and most current tutorials and documentation use this format.

Now install the two core libraries. Run npm install axios cheerio in your terminal. Axios is an HTTP client that handles sending requests to web servers, managing cookies, following redirects, and setting custom headers. Cheerio is an HTML parser that implements jQuery's selector API on the server side, letting you search through HTML documents using CSS selectors. Together, these two libraries cover the complete scraping workflow for any website that serves its content as static HTML.

Create a new file called scraper.js in your project directory. At the top, add your import statements: import axios from 'axios'; and import * as cheerio from 'cheerio';. Your project structure should now have three items: the package.json file, a node_modules directory (created by npm install), and your scraper.js file. You are ready to start writing scraping code.

Step 2: Fetch a Web Page

The first task in any scraper is downloading the HTML content of the target page. Axios makes this straightforward with its get() method. You call axios.get(url) with the URL you want to scrape, and Axios returns a promise that resolves to a response object containing the server's reply.

The response object has several useful properties. response.status contains the HTTP status code (200 means the request succeeded). response.data contains the response body, which for web pages is the raw HTML string. response.headers contains the response headers from the server. Since axios.get() returns a promise, you use await to wait for it to resolve, which means your function needs to be declared as async.

Always check the status code before proceeding with parsing. A status of 200 indicates success. A 403 means the server detected your request as automated and blocked it, which usually means you need to add a more realistic User-Agent header. A 404 means the page does not exist at that URL. A 429 means the server is rate-limiting you because you are sending requests too frequently. A 500 means the server encountered an internal error that has nothing to do with your scraper.

For more reliable requests, configure Axios with custom headers and a timeout. The most important header is User-Agent, which identifies your client to the server. Many websites block requests that do not include a browser-like User-Agent string. Set the timeout to 15000 or 30000 milliseconds (15 to 30 seconds) to prevent your scraper from hanging indefinitely on slow or unresponsive servers. You can pass these options as a second argument to axios.get() in a configuration object with headers and timeout properties.

Step 3: Parse HTML with Cheerio

Once you have the HTML content in response.data, the next step is loading it into Cheerio so you can search through it using CSS selectors. Call cheerio.load(response.data) to create a Cheerio instance. By convention, this instance is assigned to a variable named $, which mirrors the jQuery convention and makes your code immediately familiar to anyone who has used jQuery.

The $ function accepts CSS selectors and returns Cheerio objects representing the matching elements. The selector syntax is identical to what you use in CSS stylesheets and document.querySelector() in the browser. Tag selectors like $('h1') match all h1 elements. Class selectors like $('.price') match elements with the given class. ID selectors like $('#main-content') match the element with that ID. Attribute selectors like $('a[href^="https"]') match links whose href starts with https. You can combine selectors just like in CSS: $('div.product > span.price') matches span elements with class price that are direct children of div elements with class product.

Cheerio also supports chaining methods to narrow your selection. $('.product').find('.title') first selects all elements with the product class, then searches within those for elements with the title class. $('.product').first() returns only the first matching element. $('.product').eq(2) returns the third matching element (zero-indexed). $('.product').filter('.featured') narrows the selection to products that also have the featured class.

Understanding how to inspect a website's HTML structure is essential for writing effective selectors. Open the target page in Chrome or Firefox, right-click on the element containing the data you want, and select "Inspect" to open the browser's developer tools. The Elements panel shows the DOM tree with the selected element highlighted. Note the tag names, class names, and IDs of the elements surrounding your target data. These become the selectors in your Cheerio code. Look for patterns: if every product on the page is wrapped in a div with class "product-card", that class is your selector for iterating over all products.

Step 4: Extract Data from Elements

With Cheerio loaded and your selectors identified, the next step is pulling actual data values out of the HTML elements. Cheerio provides several methods for this purpose.

The .text() method returns the text content of an element, stripping all HTML tags. If the element contains child elements, .text() concatenates the text of all descendants. This is the method you use most often for extracting visible content like product names, article titles, and prices.

The .attr('name') method returns the value of a specific HTML attribute. You use this to extract href values from links ($('a').attr('href')), src values from images ($('img').attr('src')), data attributes ($('.item').attr('data-id')), and any other HTML attribute that contains data you need.

The .html() method returns the inner HTML of an element, including child tags. This is useful when you want to preserve formatting, extract embedded HTML content, or pass the content to another parser.

For scraping multiple items from a page, the .each() method lets you iterate over every element matching your selector. Inside the callback, you use $(element) to create a Cheerio object for the current element, then call .text(), .attr(), or .find() on it. Build an array before the loop, and push an object containing the extracted fields into the array on each iteration. When the loop finishes, your array contains a structured representation of every item on the page.

A practical example: if you are scraping a product listing page where each product is in a div with class "product", you would write $('.product').each((i, el) => { ... }) and inside the callback extract the title with $(el).find('.product-title').text(), the price with $(el).find('.product-price').text(), and the link with $(el).find('a').attr('href'). Push each set of values as an object into your results array.

Step 5: Handle Multiple Pages

Most real scraping tasks involve more than one page. Product catalogs span dozens or hundreds of pages. Search results are split across multiple paginated views. Job boards list positions across many category pages. Your scraper needs to handle all of these by fetching and parsing multiple URLs in sequence or in parallel.

The simplest approach is a list of known URLs. If you already know the URLs you want to scrape, store them in an array and loop through them with a for...of loop. Inside the loop, fetch each URL, parse it with Cheerio, extract the data, and push the results into your master results array. Add a delay between requests using a helper function that returns a promise wrapping setTimeout(). A delay of one to two seconds between requests is a responsible starting point that avoids overloading the target server.

For paginated websites with numbered pages, construct each page URL programmatically. If page URLs follow a pattern like /products?page=1, /products?page=2, and so on, use a for loop that increments the page number and builds the URL for each iteration. You can determine the total number of pages by checking the pagination controls on the first page, or simply loop until a page returns no results.

For sites with "next page" links rather than numbered pages, extract the next page URL from the current page after parsing it. Use a while loop that continues as long as a next page URL exists. Inside the loop, fetch the current page, extract data, find the next page link, and update your URL variable. This approach handles pagination without needing to know the total page count in advance.

When scraping many pages, concurrency control becomes important. Making hundreds of requests simultaneously can overwhelm the target server and trigger blocks. The p-limit npm package lets you run async tasks with a maximum concurrency level. For example, const limit = pLimit(3) creates a limiter that runs at most three requests concurrently. Wrap each fetch call with the limiter, and it automatically queues excess requests until a slot opens up.

Step 6: Add Error Handling and Delays

A scraper without error handling will crash the first time it encounters a network timeout, a missing element, or a rate limit response. Robust error handling is what separates a development prototype from a usable tool.

Wrap every Axios request in a try/catch block. In the catch block, check the error type. Axios errors have a response property if the server returned an HTTP error (4xx or 5xx), a code property for network-level errors like ECONNREFUSED or ETIMEDOUT, and a message property for all error types. Log the error with enough context (the URL that failed, the status code, the error message) that you can diagnose the problem later without rerunning the scraper.

Implement retry logic for transient errors. Server errors (500, 502, 503), timeouts, and connection resets are often temporary. A retry with exponential backoff (wait 1 second, then 2, then 4, then 8) gives the server time to recover. Set a maximum retry count (3 to 5 is typical) to prevent infinite loops on persistently failing URLs. Log each retry attempt so you can see the pattern in your logs.

Guard against missing elements in the HTML. Not every page may have every element you are looking for. If a product listing is missing a price, $('.price').text() returns an empty string rather than throwing an error. Check for empty strings and handle them appropriately, either by setting a default value, skipping the record, or logging a warning. The .length property of a Cheerio selection tells you how many elements matched, so if ($('.price').length === 0) lets you detect and handle missing elements explicitly.

Add delays between requests to be a responsible scraper. Create a simple delay function: const delay = (ms) => new Promise(resolve => setTimeout(resolve, ms));. Call await delay(1500) between requests to add a 1.5-second pause. For added stealth, randomize the delay: await delay(1000 + Math.random() * 2000) waits between 1 and 3 seconds, which looks more like human browsing behavior than a fixed interval.

Step 7: Save Results to a File

The final step is persisting your extracted data to a file that you can use for analysis, import into a database, or share with colleagues.

For JSON output, use Node's built-in fs module. Import it with import fs from 'fs';, then call fs.writeFileSync('results.json', JSON.stringify(results, null, 2)). The second argument to JSON.stringify (null) skips any custom replacer, and the third argument (2) adds two-space indentation for readability. JSON is the natural choice for JavaScript projects because it preserves data types, handles nested structures, and is easy to parse back into JavaScript objects.

For CSV output, install the json2csv package with npm install json2csv. Import the Parser class and create a new instance, optionally specifying which fields to include. Call parser.parse(results) to convert your array of objects to a CSV string, then write it to a file with fs.writeFileSync('results.csv', csv). CSV is the best format when you need to share data with people who will open it in Excel or Google Sheets, or when your data is flat and tabular without nested structures.

For large-scale scraping where holding all results in memory is impractical, use streaming writes. Node's fs.createWriteStream() creates a writable stream that you can append to incrementally. For JSON, write one object per line (JSON Lines format) by calling stream.write(JSON.stringify(item) + '\n') for each scraped item. For CSV, write the header row first, then append one row per item. Streaming keeps memory usage constant regardless of how many items you scrape.

After writing your file, log a summary: the number of items scraped, the number of errors encountered, and the output file path. This gives you a quick confirmation that the scraper ran successfully and produced the expected volume of data.