Web Scraping with Cheerio

Step 1: Install and Import Cheerio

Cheerio is an npm package that you install into any Node.js project. Start by creating a project directory and initializing it with npm init -y. Then install Cheerio and an HTTP client with npm install cheerio axios. Axios handles fetching the HTML from web servers, while Cheerio handles parsing that HTML and extracting data from it.

In your JavaScript file, import both libraries. If you are using ES modules (recommended for new projects), add "type": "module" to your package.json, then use import axios from 'axios'; and import * as cheerio from 'cheerio';. If you are using CommonJS, use const axios = require('axios'); and const cheerio = require('cheerio');. Both module systems work identically with Cheerio.

Cheerio version 1.0, released in late 2023, introduced some API changes from the older 0.x versions. The most notable change is that cheerio.load() is now the primary way to create a Cheerio instance. Older tutorials that use $.load() or other deprecated methods may need adjustment. This guide uses the current 1.x API throughout.

Step 2: Fetch and Load HTML

Before Cheerio can parse any HTML, you need to download the page content from the web. Axios's get() method sends an HTTP GET request and returns the HTML in response.data. Wrap the call in an async function and use await to wait for the response.

Once you have the HTML string, pass it to cheerio.load() to create a Cheerio instance. By convention, this instance is assigned to a variable named $, which mirrors jQuery's convention. The $ function is now ready to accept CSS selectors and return matching elements from the parsed HTML document.

The cheerio.load() function accepts optional configuration. The second argument is an options object where you can set xml: true to parse the input as XML rather than HTML, or decodeEntities: false to preserve HTML entities in their encoded form rather than decoding them to their character equivalents. For most scraping tasks, the defaults work fine and you do not need to pass any options.

A common mistake is trying to load a URL directly into Cheerio. Cheerio does not fetch pages, it only parses HTML strings. You must always fetch the page with an HTTP client first, then pass the response body to Cheerio. This separation of concerns is actually an advantage because it gives you full control over the HTTP request, including headers, cookies, proxy configuration, and retry logic, before Cheerio ever touches the HTML.

Step 3: Select Elements with CSS Selectors

Cheerio's selector engine supports the same CSS selector syntax you use in stylesheets and document.querySelector(). This means you can target elements by tag name, class, ID, attribute, pseudo-class, and complex combinators without learning a new query language.

Tag selectors match elements by their HTML tag name. $('h1') selects all h1 elements, $('p') selects all paragraphs, and $('a') selects all links. Class selectors use the dot prefix: $('.product-name') matches elements with the class "product-name". ID selectors use the hash prefix: $('#main-content') matches the single element with that ID. You can combine these: $('div.product') matches div elements with the product class.

Attribute selectors target elements based on their HTML attributes. $('a[href]') matches all links that have an href attribute. $('a[href^="https"]') matches links whose href starts with "https". $('input[type="text"]') matches text input fields. $('[data-id]') matches any element with a data-id attribute. Attribute selectors are particularly useful for scraping because data attributes and specific attribute values are often more stable than class names, which may change during site redesigns.

Combinator selectors describe relationships between elements. $('div.product > h3') selects h3 elements that are direct children of div.product. $('.sidebar a') selects links anywhere inside an element with class sidebar (descendant combinator). $('h2 + p') selects the paragraph immediately following an h2 (adjacent sibling). $('h2 ~ p') selects all paragraphs that are siblings of an h2 (general sibling).

Pseudo-class selectors add filtering power. $('.item:first-child') selects the first .item element among its siblings. $('.item:nth-child(3)') selects the third child. $('.item:not(.sold-out)') selects items that do not have the sold-out class. $('tr:nth-child(even)') selects even-numbered table rows, useful for scraping tables that use alternating row styles.

Step 4: Extract Text and Attributes

Once you have selected the right elements, you need to extract their data. Cheerio provides three primary extraction methods: .text(), .attr(), and .html().

The .text() method returns the combined text content of the element and all its descendants, with HTML tags stripped. For example, if a div contains <span>Price: <strong>$29.99</strong></span>, calling .text() on the div returns "Price: $29.99" as a plain string. This is the method you will use most often for extracting visible content like product names, prices, article titles, and descriptions. Note that .text() on a selection of multiple elements concatenates all their text, which is usually not what you want. Call .text() on individual elements, not on collections.

The .attr('name') method returns the value of a specific HTML attribute. $('a').attr('href') returns the link's URL. $('img').attr('src') returns the image's source URL. $('.product').attr('data-price') returns a data attribute value. If the attribute does not exist on the element, .attr() returns undefined. When called on a collection of multiple elements, .attr() returns the attribute value from only the first element in the collection.

The .html() method returns the inner HTML of an element, including child tags. This is useful when you need to preserve formatting (bold, italic, links) within extracted content, or when you want to extract a chunk of HTML for further processing. For most data extraction tasks, .text() and .attr() are sufficient.

Text cleaning is an important step after extraction. Web page text often contains extra whitespace, newlines, and tab characters that do not appear visually in the browser but are present in the HTML source. Use .text().trim() to remove leading and trailing whitespace. For more aggressive cleaning, use a regular expression to collapse multiple whitespace characters into a single space: .text().replace(/\s+/g, ' ').trim(). Price strings often need cleaning too: extracting "$29.99" as a number requires removing the currency symbol and parsing the result with parseFloat().

Step 5: Iterate Over Multiple Elements

Most scraping tasks involve extracting data from multiple similar elements on a page, such as a list of products, a table of search results, or a series of article cards. Cheerio provides .each() and .map() methods for iterating over collections of matching elements.

The .each() method executes a callback function for every element in the selection. The callback receives the index and the raw DOM element as arguments. Inside the callback, wrap the element with $(element) to get a Cheerio object that you can call .text(), .attr(), and .find() on. A typical pattern is to create an empty array before the loop, then push an object with extracted fields into the array on each iteration.

The .map() method works similarly but returns a Cheerio object containing the mapped values. Call .get() on the result to convert it to a plain JavaScript array. The .map() approach is more concise for simple extractions where each element maps to a single value, like extracting all link URLs: $('a').map((i, el) => $(el).attr('href')).get() returns an array of href strings.

When extracting structured data from a list of elements, the pattern is always the same. First, select the container element that wraps each item (the product card, the table row, the list item). Second, iterate with .each(). Third, inside the callback, use .find() to locate child elements within the current item. Fourth, extract values with .text() and .attr(). Fifth, build an object from the extracted values and push it to your results array. This pattern handles everything from simple link lists to complex product catalogs.

For tables, the approach is slightly different. Select all tr elements within the table body ($('table tbody tr')), then iterate over them. Inside each row, select all td elements and extract their text values. You can map the values to field names by position (first td is name, second is price, third is quantity) or by using the header row to determine field names dynamically.

Step 6: Handle Tables and Nested Structures

Real-world HTML is not always cleanly structured. Data is embedded in deeply nested divs, split across sibling elements, hidden in data attributes, or organized in tables with merged cells and irregular layouts. Cheerio's DOM traversal methods handle these scenarios.

The .find(selector) method searches within the current element's descendants for elements matching the selector. This is how you drill into nested structures: $('.product').find('.price-container .current-price') finds the current price element nested several levels deep inside each product element. .find() is the most commonly used traversal method and handles most nested data extraction needs.

The .children(selector) method is similar to .find() but only looks at direct children, not all descendants. This is useful when the same class name appears at multiple nesting levels and you need to distinguish between them. .parent() moves up one level in the DOM tree, and .parents(selector) moves up through all ancestors until it finds one matching the selector.

The .siblings(selector) method selects elements that share the same parent as the current element. .next() and .prev() select the immediately following and preceding sibling elements, respectively. These are useful when related data is split across sibling elements, for example a dt/dd pattern in a definition list where the label and value are siblings rather than parent-child.

For HTML tables, a robust scraping approach reads the header row to determine column names, then maps each data row's cells to those column names. Select the header cells with $('table thead th') and extract their text into an array of column names. Then iterate over body rows with $('table tbody tr').each(), and inside each row, select all td elements and map each cell's text to the corresponding column name by index. This produces an array of objects where each object represents one row with named fields, regardless of column order changes.

Tables with merged cells (colspan or rowspan attributes) are more complex. You need to track which columns are occupied by spans from previous rows and adjust your cell-to-column mapping accordingly. For tables this complex, consider whether the data is available through an API endpoint instead, which almost always provides cleaner data than parsing a complex HTML table.