Web Scraping with Axios and Cheerio

Step 1: Configure Axios for Scraping

Raw axios.get() calls work for quick tests, but production scrapers need a configured Axios instance with sensible defaults for headers, timeouts, and error handling. Creating a reusable instance centralizes this configuration so every request inherits the same settings.

Create an Axios instance with axios.create() and pass a configuration object. Set the timeout to 20000 milliseconds (20 seconds), which is long enough for slow servers but short enough to prevent your scraper from hanging indefinitely. Set maxRedirects to 5, which handles typical redirect chains without following infinite redirect loops. Set validateStatus to a function that accepts status codes in the 200 range, so Axios throws errors for 4xx and 5xx responses that you can catch and handle explicitly.

The headers configuration is critical for avoiding blocks. At minimum, set a realistic User-Agent string from a current Chrome or Firefox release. Many servers also check the Accept, Accept-Language, and Accept-Encoding headers. A complete header set that matches a real Chrome browser includes Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8, Accept-Language: en-US,en;q=0.9, and Accept-Encoding: gzip, deflate, br. Axios handles gzip decompression automatically when Accept-Encoding includes gzip.

Use Axios interceptors to add dynamic behavior to all requests. A request interceptor can rotate User-Agent strings from a pool on every request, add a random delay before each request, or inject authentication tokens. A response interceptor can log response times for performance monitoring, detect rate limiting responses, or automatically retry failed requests. Interceptors keep this logic in one place rather than duplicating it at every fetch call in your scraper.

Step 2: Fetch Pages with Custom Headers

Each request to a target server should mimic a real browser visit closely enough to avoid triggering bot detection. Beyond the default headers set on your Axios instance, some requests may need additional customization.

The Referer header tells the server which page the request originated from. When scraping a product detail page, setting the Referer to the category page that links to it makes the request look more natural. Some servers block requests without a Referer header entirely. Set it to the logical parent page or the target site's homepage.

For sites that serve different content based on geography, the Accept-Language header controls which language version you receive. Set it to match the language of the content you want to scrape. Some sites also check the request's IP address for geographic targeting, which is where proxy services with location-specific exit nodes become useful.

POST requests are necessary for scraping sites that use form submissions for search, filtering, or pagination. Axios's post() method accepts a data payload as the second argument. For form submissions, send the data as URL-encoded parameters with Content-Type: application/x-www-form-urlencoded. For API endpoints, send JSON with Content-Type: application/json. Inspect the target site's form submissions in your browser's Network tab to determine the correct content type and required form fields.

Handle redirects carefully. Axios follows redirects automatically by default, but some sites use redirect chains for tracking or session management. If you need to capture intermediate redirect URLs, set maxRedirects: 0 and handle the 301/302 response manually by reading the Location header. This is useful for scraping sites that redirect to login pages when authentication is missing, letting you detect the redirect and handle authentication before retrying.

Step 3: Parse Responses with Cheerio

After Axios fetches a page and returns the HTML in response.data, pass it to cheerio.load() to create a parseable DOM. The resulting $ function works identically to the one described in our Cheerio guide, accepting CSS selectors and returning Cheerio objects.

Build a dedicated parsing function for each page type. The function takes the HTML string and the page URL as inputs, creates a Cheerio instance, extracts all target fields, cleans and validates the data, and returns a structured object or array. Passing the page URL as a parameter lets the function resolve relative URLs to absolute ones using the URL constructor.

A robust parsing function validates its output before returning. Check that required fields are non-empty strings, that numeric fields parse correctly with parseFloat() or parseInt(), and that URLs are well-formed. If critical fields are missing, throw a descriptive error or return a null value that the calling code can handle. This validation catches selector breakage (from target site redesigns) immediately rather than letting bad data propagate silently into your dataset.

For pages with multiple data items (product listings, search results, directory entries), the pattern is to select all container elements with $('.item-class'), iterate with .each() or .map(), and extract fields from each container using .find(). Build an array of objects, one per item, and return the complete array. This pattern handles zero items (empty search results) gracefully because .each() on an empty selection simply does not execute.

Step 4: Handle Cookies and Sessions

Many websites set cookies on the first visit and expect subsequent requests to include those cookies. Without cookie handling, each request looks like a new visitor, which can trigger additional security checks, change the content served, or break multi-step workflows like search-then-paginate.

Axios does not manage cookies automatically across requests. To maintain cookies, use the tough-cookie package along with axios-cookiejar-support. Install both packages, then wrap your Axios instance with wrapper(axiosInstance) and provide a CookieJar instance in the request configuration. With this setup, cookies set by response headers are automatically stored in the jar and sent with subsequent requests, mimicking how a real browser maintains session state.

For scraping behind a login, follow this sequence: first, send a GET request to the login page to receive any CSRF tokens or initial cookies. Then send a POST request to the login endpoint with your credentials and any required tokens. The cookie jar automatically captures the authentication cookies from the login response. All subsequent requests include these cookies and are treated as authenticated by the server.

If you need to scrape with multiple user sessions simultaneously (for parallel scraping across different accounts or regions), create a separate Axios instance with its own cookie jar for each session. This keeps sessions isolated so cookies from one session do not leak into another.

Some websites use JavaScript-set cookies (via document.cookie) rather than HTTP response headers. Axios cannot capture these because it does not execute JavaScript. If you encounter a site where cookie-based authentication or session management depends on client-side JavaScript, you need to switch to Puppeteer or Playwright for those specific requests, or manually set the cookies in your Axios instance after determining their values through browser inspection.

Step 5: Scrape Multiple Pages with Concurrency

Real scraping projects involve fetching dozens to thousands of pages. Doing this one page at a time is painfully slow, but fetching them all simultaneously will get you blocked instantly. Controlled concurrency lets you run multiple requests in parallel while staying within limits the target server will tolerate.

The p-limit package provides the simplest concurrency control. Import it, create a limiter with your desired concurrency level (start with 3 to 5), and wrap each fetch-and-parse operation with the limiter function. Collect all the wrapped promises in an array and pass them to Promise.allSettled(). Using allSettled instead of all ensures that one failed request does not abort the entire scrape, because allSettled waits for all promises to complete regardless of individual failures.

Add delays between requests even with concurrency limits. Inside your fetch function, call your delay helper after each request completes. A randomized delay between 500ms and 2000ms prevents uniform request timing, which is one of the most obvious signals of automated traffic. The delay function is a one-liner: const delay = (ms) => new Promise(r => setTimeout(r, ms));.

For two-level scraping (listing page followed by detail pages), process the listing pages first to collect detail page URLs, then process all detail pages with concurrency control. This is more efficient than processing each listing page and its detail pages sequentially, because it maximizes utilization of your concurrency slots.

Monitor progress during long scrapes. Log the number of pages processed and remaining at regular intervals. Track the success rate (successful requests divided by total attempts) and automatically reduce concurrency if the rate drops below a threshold, which usually indicates the target server is struggling or starting to block requests. This adaptive throttling keeps your scraper running smoothly without manual intervention.

Step 6: Export Scraped Data

The final step is saving your extracted data in a format that serves your downstream workflow. The right format depends on data structure, volume, and how the data will be consumed.

For JSON output, use Node's built-in fs module. For small datasets (under 10,000 records), fs.writeFileSync('output.json', JSON.stringify(results, null, 2)) writes the entire dataset at once with readable formatting. For larger datasets, use JSON Lines format (one JSON object per line) with streaming writes. Create a write stream with fs.createWriteStream('output.jsonl') and call stream.write(JSON.stringify(item) + '\n') for each record. JSON Lines is appendable, streamable, and uses constant memory regardless of dataset size.

For CSV output, the json2csv package converts arrays of objects to CSV strings. Import Parser from @json2csv/plainjs, create a parser with your field definitions, and call parser.parse(results). Write the CSV string to a file with fs.writeFileSync(). For streaming CSV export during a scrape, use @json2csv/node's Transform stream to convert objects to CSV rows as they are produced.

For database storage, better-sqlite3 provides synchronous, high-performance SQLite access that works well for scraping because scrapers typically write records sequentially. Create a table matching your data schema, prepare an INSERT statement, and call it for each scraped record. Use transactions to batch inserts for better performance: begin a transaction, insert 100 to 1000 records, then commit. This reduces disk I/O and speeds up insertion by an order of magnitude compared to individual inserts.

Always log a summary after the scrape completes: total pages processed, total items extracted, total errors, and the output file path or database location. This summary confirms the scrape ran successfully and produced the expected volume of data.