How to Handle Pagination in JavaScript Web Scraping

Step 1: Identify the Pagination Pattern

Before writing any pagination code, examine the target website to determine which pattern it uses. Open the page in your browser and look at the bottom of the results list for pagination controls.

Numbered page links show page numbers as clickable links, typically "1 2 3 ... 50" or similar. The URLs usually include a query parameter like ?page=2 or ?p=3 or a path segment like /page/2. This is the most common pagination pattern on traditional websites and the easiest to scrape.

Next-page links show "Next" or an arrow button without numbered pages. The URL for the next page is in the link's href attribute. You follow these links one at a time until no next link exists on the page.

Infinite scroll loads new content automatically when you scroll to the bottom of the page. There are no visible pagination controls. This pattern is common on social media feeds, image galleries, and modern search results. It requires browser automation because scrolling triggers JavaScript that fetches and inserts new content.

Load-more buttons show a "Load More" or "Show More" button at the bottom of the results. Clicking the button triggers a JavaScript call that fetches and appends new items to the existing list. Like infinite scroll, this requires browser automation to trigger the load action.

API pagination uses backend API endpoints with offset/limit or cursor-based parameters. Check the Network tab in your browser's developer tools to see if the page fetches data from an API. If it does, you can often call the API directly with Axios, bypassing the HTML entirely. API responses typically include metadata about total results and how to get the next page.

To identify the pattern, check both the visible pagination controls and the Network tab. Sometimes a site appears to use one pattern visually but actually loads data through API calls behind the scenes. If you find API calls, use them directly, because they are faster and produce cleaner data than HTML parsing.

Step 2: Scrape Numbered Page URLs

Numbered pagination is the simplest pattern to scrape because the URL structure is predictable. If the first page is /products?page=1, then page two is /products?page=2, page three is /products?page=3, and so on. You construct each URL programmatically and fetch them in a loop.

The first decision is how to determine the total number of pages. Three approaches work: extract the total page count from the pagination controls on the first page (look for "Page 1 of 47" text or the last numbered link), extract the total item count and divide by items per page, or simply loop until a page returns zero results. The third approach is the most robust because it does not depend on specific HTML elements that might change during a site redesign.

Implement the loop as a for loop when you know the total pages, or a while loop with a results check when you do not. Inside the loop, construct the page URL by substituting the page number into the URL template. Fetch the page with Axios, parse with Cheerio, extract items, and push them to your results array. Add a delay after each request to avoid overwhelming the server.

Numbered pagination can also be parallelized with concurrency control. If you know the total number of pages, generate all page URLs upfront and process them concurrently using p-limit. This dramatically speeds up the scrape compared to sequential processing. However, some sites block parallel requests to their paginated listings more aggressively than sequential ones, so monitor your error rate and reduce concurrency if needed.

Watch for URL parameter variations. Some sites use page, others use p, pg, pageNumber, or offset (where offset equals page number times items per page). Some use zero-indexed pages (first page is 0) while others use one-indexed pages (first page is 1). Check the actual URLs in the pagination links to determine the exact parameter name, starting value, and increment.

Step 3: Follow Next-Page Links

When a site shows only a "Next" link without numbered pages, you cannot predict the URL structure. Instead, you extract the next page URL from the current page and follow it iteratively.

The pattern is a while loop. Start with the first page URL. Inside the loop, fetch the page, parse it with Cheerio, extract the data items, and then look for the next page link. Common selectors for the next link include a.next, a[rel="next"], .pagination a:last-child, or an element containing the text "Next". Extract the href attribute from this element. If the href exists and is not empty, resolve it to an absolute URL (using new URL(href, currentUrl).href) and set it as the URL for the next iteration. If no next link exists, break out of the loop.

Guard against infinite loops. Some sites have pagination bugs where the "next" link on the last page points back to the first page or to the current page. Maintain a Set of URLs you have already visited. Before fetching a URL, check if it is already in the set. If it is, break out of the loop. This prevents your scraper from cycling through pages indefinitely.

Also set a maximum page count as a safety limit. Even with deduplication, unexpected pagination behavior can cause your scraper to run forever. A maximum of 1,000 or 10,000 pages (depending on the expected dataset size) provides a reasonable safety net.

Step 4: Handle Infinite Scroll

Infinite scroll requires browser automation because the pagination is triggered by JavaScript scroll events, not by navigating to a new URL. Puppeteer or Playwright loads the page in a real browser, and your scraper programmatically scrolls to the bottom of the page to trigger new content loading.

The basic infinite scroll pattern works like this: navigate to the page and wait for initial content to load. Count the number of items currently on the page. Scroll to the bottom with page.evaluate(() => window.scrollTo(0, document.body.scrollHeight)). Wait for new content to appear using page.waitForFunction() that checks whether the item count has increased. If new items appeared, repeat the scroll. If the item count did not change after waiting (typically 3 to 5 seconds), assume you have reached the end and stop scrolling.

Use a page.waitForFunction() that is smarter than just waiting a fixed time. Pass the previous item count as a parameter and wait until the current count exceeds it: page.waitForFunction((prevCount) => document.querySelectorAll('.item').length > prevCount, {}, previousCount). Set a timeout on this wait (5 to 10 seconds). If it times out, you have reached the end of the data.

Some infinite scroll implementations load new content from an API endpoint that you can see in the Network tab. If you identify this endpoint, you can skip the browser entirely and call the API directly with Axios. The API usually accepts a pagination parameter (page number, offset, or cursor token) that you increment with each request. This approach is significantly faster and more reliable than actually scrolling a browser.

For very long feeds (thousands of items), scrolling can become slow and memory-intensive as the browser accumulates items in the DOM. Extract and save data periodically during the scroll (every 50 to 100 items) rather than waiting until the end. You can also remove already-processed items from the DOM using page.evaluate() to keep memory usage manageable during long scrolling sessions.

Step 5: Handle Load-More Buttons

Load-more buttons function similarly to infinite scroll but require a click instead of a scroll event. The button triggers a JavaScript call that fetches and appends new items to the existing list.

The pattern is: navigate to the page, wait for the initial content and the load-more button to appear, extract data from the current items, click the button with page.click('.load-more-button'), wait for new items to load (using page.waitForFunction() to check that the item count increased), and repeat until the button disappears or no new items load.

The button selector may change between states. Some sites disable the button on the last page, change its text from "Load More" to "No More Results", or remove it from the DOM entirely. Check for the button's presence before attempting to click it: const hasMore = await page.$('.load-more-button'). If the element does not exist, exit the loop.

Like infinite scroll, check the Network tab for the API call triggered by the load-more button. If you can identify the endpoint and its pagination parameters, calling the API directly with Axios eliminates the need for browser automation entirely. The API approach is always preferable when available.

Step 6: Paginate Through APIs

API pagination is the fastest, most reliable, and cleanest pagination approach because you get structured JSON data directly from the server without any HTML parsing. Many modern websites, even those that display traditional-looking pagination controls, actually load their data from backend APIs that you can call directly.

Offset-based APIs use offset and limit parameters. The first request uses offset=0&limit=20 to get items 1 through 20. The next request uses offset=20&limit=20 for items 21 through 40. Continue incrementing the offset by the limit until the response contains fewer items than the limit, which indicates you have reached the last page. The total item count is often included in the response metadata, letting you calculate the total number of requests needed.

Cursor-based APIs use a token or ID that points to the next batch of results. The first request has no cursor. The response includes a nextCursor or nextPageToken field. Pass this token as a parameter in the next request. Continue until the response does not include a next cursor, indicating the last page. Cursor pagination is more robust than offset pagination because inserting or deleting items between requests does not cause you to skip or duplicate records.

Page-number APIs use a simple page parameter, starting at 1 or 0. Each response may include a totalPages or hasNextPage field. Loop through pages until you reach the total or until hasNextPage is false.

API pagination with Axios is straightforward. Make a GET request with the pagination parameters, parse the JSON response (Axios handles this automatically), extract the data items and the next-page indicator, and loop until done. No Cheerio parsing is needed because the data is already structured. This approach typically runs 10 to 50 times faster than HTML scraping with pagination.

Rate limits are more strictly enforced on API endpoints than on page loads, so add delays between API requests. Watch for X-RateLimit-Remaining and X-RateLimit-Reset response headers that tell you exactly how many requests you have left and when the limit resets.