How to Scrape Paginated Sites in Python

Step 1: Identify the Pagination Pattern

Before writing any pagination code, examine how the target site structures its pages. Open the site in your browser and navigate through several pages while watching the URL and the page content. There are four common patterns:

Next-page links: The page contains a link (usually labeled "Next" or represented by an arrow icon) that points to the next page. The URL often includes a page number or offset parameter. This is the most common pattern on blogs, forums, and news sites.

Numbered page parameters: The URL contains a page parameter like ?page=1, ?page=2, or an offset parameter like ?offset=0, ?offset=20. Each page shows a fixed number of results. Clicking page numbers in the pagination bar updates only this parameter. This is standard on e-commerce sites and search results.

API-based pagination: The page's JavaScript calls a backend API that returns data in JSON format. The API uses pagination parameters like offset and limit, or cursor-based pagination with a token that points to the next batch. This pattern is found on modern single-page applications.

Infinite scroll: There are no page links or numbers at all. New content loads automatically when you scroll to the bottom of the page. The loading is triggered by JavaScript, which makes AJAX calls to fetch more content. This pattern is common on social media platforms and image galleries.

Check the browser's developer tools (Network tab) while navigating between pages. If you see XHR or Fetch requests firing when you change pages, the site may be using API-based pagination, which is often easier to scrape than HTML pagination.

Step 2: Scrape Next-Page Link Pagination

Next-page link pagination is the simplest to handle. The logic is a while loop that processes the current page, finds the next-page link, and follows it until no next link is found.

Start by fetching the first page and parsing it with BeautifulSoup. Extract the data you want from the current page, then look for the next-page link using a selector that targets the pagination element. Common selectors include a.next, a[rel="next"], li.next a, or pagination links containing text like "Next" or a right arrow character.

When you find the next-page link, extract its href attribute. The href may be a relative URL (like /products?page=2) or an absolute URL. Use urllib.parse.urljoin(current_url, href) to resolve relative URLs to absolute ones. This function correctly handles all relative URL formats, including paths starting with /, paths without a leading slash, and query-string-only URLs.

If the next-page link is not found (the selector returns None), you have reached the last page and the loop should stop. Always add a delay between requests using time.sleep() to avoid overwhelming the server. A random delay between 1 and 3 seconds is a reasonable default.

Keep a count of pages processed and set a maximum page limit as a safety measure. If your scraper encounters a bug in its next-link detection (for example, if the selector accidentally matches a different link), the maximum page limit prevents an infinite loop that would hammer the server with requests.

Step 3: Handle Numbered Page Parameters

When pagination uses URL parameters, you can build each page's URL programmatically without parsing any links from the HTML. This approach is faster and more reliable than following next-page links because you do not depend on specific HTML elements.

The most common parameter format is ?page=N where N starts at 1. A simple for loop or while loop that increments the page number and constructs the URL handles this directly. For offset-based pagination like ?offset=0&limit=20, increment the offset by the limit value on each iteration.

Determine the total number of pages or items before starting the loop if possible. Many sites display "Page 1 of 25" or "Showing 1-20 of 500 results" on the first page. Extract this total from the first page's HTML and calculate how many pages you need to fetch. This lets you use a for loop with a known range instead of a while loop, and it gives you accurate progress tracking.

If the total is not available, use a while loop with stop conditions. When a page returns zero results, contains the same items as the previous page, or returns a 404 error, you have passed the last page. Always check the response status code and the number of extracted items on each iteration.

Some sites use non-standard parameter names like ?p=2, ?start=20, or ?cursor=abc123. Examine the URL structure carefully on the first few page transitions to identify the correct parameter name and increment pattern.

Step 4: Work with API-Based Pagination

API pagination is often the most efficient approach because you get structured JSON data instead of HTML that needs parsing. Use your browser's developer tools to identify the API endpoints that the site's JavaScript calls to load paginated data.

Offset/limit APIs accept two parameters: an offset (the number of items to skip) and a limit (the number of items to return). Start with offset=0 and increment by the limit value on each request. Stop when the response returns fewer items than the limit, which indicates you have reached the last page.

Cursor-based APIs return a cursor token with each response that you pass in the next request to get the following batch. This pattern is more common in modern APIs because it handles concurrent insertions and deletions better than offset-based pagination. The response typically includes a field like next_cursor or has_more that tells you whether more pages exist. When next_cursor is null or has_more is false, you have collected all the data.

API responses often include metadata that tells you the total number of items available, the current page, and the total number of pages. Use this information for progress tracking and to set up a loop with a known endpoint.

When calling APIs directly, copy the required headers from the browser's developer tools. Many APIs require specific headers like an Accept header with a JSON content type, an authorization token, or a custom header that identifies the client. Without these headers, the API may return errors or empty results.

Step 5: Scrape Infinite Scroll Pages

Infinite scroll requires browser automation because the loading is triggered by JavaScript scroll events. Playwright is the standard tool for this in Python.

The basic approach is a loop that scrolls to the bottom of the page, waits for new content to load, and checks whether new items appeared. Use Playwright's JavaScript evaluation to scroll: page.evaluate("window.scrollTo(0, document.body.scrollHeight)"). After each scroll, wait for either new elements to appear (using page.wait_for_selector) or a timeout (using page.wait_for_timeout) to give the server time to respond.

Track the number of items on the page before and after each scroll. If the count does not increase after a scroll and a reasonable wait, you have reached the end of the available content. Some sites show a "No more results" message at the bottom, which you can also check for as a stop condition.

For better performance, intercept the API calls that the infinite scroll triggers rather than parsing the rendered DOM. Many infinite scroll implementations call a REST API endpoint that returns a batch of items in JSON. If you identify this endpoint, you can call it directly with incrementing offset parameters, bypassing the browser entirely and converting the problem into Step 4 above.

Set a maximum scroll count to prevent infinite loops. Social media feeds, for example, can have effectively unlimited content. Decide how much data you need before starting, and stop scrolling once you reach that threshold.

Step 6: Detect the Last Page

Reliable last-page detection prevents both premature stopping (missing data) and infinite loops (wasting resources and potentially getting blocked). Use multiple stop conditions together for robustness.

The primary condition is content-based: if the current page returns zero new items, you have passed the last page. Compare item counts or item identifiers (like URLs or IDs) between pages to detect this. Some sites return the first page's content when you request a page number beyond the maximum, so checking for duplicate content is important.

The secondary condition is HTTP-based: if the server returns a 404 status code or a redirect to the first page, you have exceeded the valid page range. Always check response.status_code before parsing.

The safety condition is a hard maximum: set a maximum number of pages (for example, 100 or 500) that your scraper will process regardless of other conditions. This prevents runaway scrapers in case of bugs in your detection logic. Log a warning when the maximum is reached so you know whether you need to increase it.

For API-based pagination, use the metadata provided in the response. If the API returns a total_count or has_next_page field, these are the most reliable indicators. For cursor-based APIs, a null cursor always means the end of the dataset.

When scraping incrementally (adding new data to an existing dataset), consider stopping when you encounter items you have already collected. This is efficient for scrapers that run regularly, such as daily price checks, because they only need to collect new items since the last run rather than re-scraping the entire dataset.