Scraping Dynamic Pages with Python
If you have tried scraping a website with Requests and BeautifulSoup and found that the data you want is missing from the HTML response, you are probably dealing with a dynamic page. The HTML returned by the server contains only a shell, and JavaScript fills in the content after the page loads in a browser. Single-page applications built with React, Vue, or Angular are almost entirely dynamic. E-commerce product listings, social media feeds, and search results often load data through background API calls as well.
Before jumping to browser automation, it is worth checking whether the data is available through a simpler method. The steps below are ordered from simplest to most complex, so you can start with the easiest approach and escalate only when needed.
Step 1: Identify Dynamic Content
The first step is confirming that the page actually renders content dynamically. Open the target URL in your browser and view the page source (Ctrl+U or Cmd+U). This shows the raw HTML that the server sends, before any JavaScript executes. Compare it with what you see on the rendered page.
If the data you want appears in the page source, the content is static and you can scrape it with Requests and BeautifulSoup. No browser automation needed. If the page source contains empty containers, placeholder text, or JavaScript framework boilerplate instead of the actual content, the page is dynamic.
Another quick test is to fetch the page with Requests and search the response text for a known value that appears on the rendered page. If the value is present in the response, the page is static for your purposes. If the value is absent, JavaScript is loading it after the initial response.
Not all dynamic content requires a headless browser. Many dynamic pages load data from backend API endpoints that return structured JSON. If you can identify and call these endpoints directly, you avoid the overhead, complexity, and slowness of browser automation entirely.
Step 2: Check for Hidden APIs
Open your browser's developer tools and switch to the Network tab. Reload the page and watch the network requests that fire. Filter by "XHR" or "Fetch" to isolate the JavaScript-initiated API calls. These are the requests that the page's JavaScript code makes to fetch data from the server.
Click on individual requests to inspect their URLs, request headers, query parameters, and response bodies. Many sites have well-structured REST APIs or GraphQL endpoints that return the exact data displayed on the page, but in clean JSON format rather than HTML. An e-commerce product listing page might make a call to something like /api/products?category=electronics&page=1 that returns a JSON array of products with all their fields.
If you find such an API endpoint, you can call it directly with Requests or httpx. Copy the URL and any required headers (authentication tokens, API keys, or session cookies) from the developer tools. Build your scraper around these API calls instead of parsing HTML. This approach is faster, more reliable, and uses far fewer resources than launching a browser.
Be aware that some APIs require specific headers, cookies, or tokens that are set by the initial page load. In those cases, you may need to first fetch the main page to obtain the required tokens, then use those tokens in your API calls. A Requests Session handles this naturally by persisting cookies across requests.
Step 3: Set Up Playwright for Browser Scraping
When direct API calls are not feasible, you need a tool that can run a real browser. Playwright is the current standard for this in Python. Install it with pip install playwright, then run playwright install to download the browser binaries for Chromium, Firefox, and WebKit.
Playwright provides both synchronous and asynchronous APIs. The synchronous API is simpler for basic scraping scripts. You create a Playwright instance, launch a browser (typically Chromium in headless mode), open a new page, and navigate to your target URL.
Headless mode runs the browser without a visible window, which is faster and suitable for production scraping. During development, you can set headless=False to see the browser window, which is invaluable for debugging selectors and understanding page behavior.
Playwright's page.goto(url) navigates to a URL and waits for the page to reach a "load" state by default. However, "load" only means the initial HTML and resources are loaded, not that all JavaScript has finished executing. For dynamic content, you need more specific wait conditions, which the next step covers.
For a comprehensive introduction to Playwright's features, see our Playwright guide.
Step 4: Wait for Elements and Extract Data
The most common cause of failed dynamic scraping is trying to extract data before JavaScript has finished rendering it. Playwright solves this with explicit wait conditions that pause execution until specific criteria are met.
The page.wait_for_selector(selector) method pauses until an element matching the CSS selector appears in the DOM. This is the most commonly used wait condition. For example, if product data loads into elements with the class "product-card", calling page.wait_for_selector(".product-card") ensures you do not try to extract data before it exists.
The page.wait_for_load_state("networkidle") method waits until there have been no network requests for at least 500 milliseconds. This is useful when you do not know the exact selector to wait for but want to ensure all API calls have completed. Be cautious with this on pages that have continuous background activity (analytics, ads, websockets), as "networkidle" may never trigger.
Once the content is loaded, extract data using Playwright's locator API or by passing the rendered HTML to BeautifulSoup. Playwright's page.query_selector_all(selector) returns elements that you can call .text_content() or .get_attribute(name) on. Alternatively, grab the full rendered HTML with page.content() and parse it with BeautifulSoup, which lets you reuse your existing parsing code.
Playwright locators also support actions like clicking buttons, filling form fields, and selecting dropdown options. This is useful for sites that require interaction before revealing content, such as clicking a "Show More" button or selecting a filter category.
Step 5: Intercept Network Requests
One of Playwright's most powerful features for scraping is network request interception. Instead of parsing the rendered HTML, you can capture the API responses that the page's JavaScript uses to populate its content. This gives you clean, structured data without navigating the DOM.
Use page.on("response", handler) to register a callback that fires whenever the browser receives a response. In the handler, check the response URL to identify the API calls you care about. When you find a matching response, call response.json() to parse the JSON body directly.
The page.route(url_pattern, handler) method gives even more control. You can modify requests before they are sent (changing headers or parameters), block specific requests (like ads and tracking scripts to speed up page loads), or redirect requests to different URLs. Blocking unnecessary resources like images, stylesheets, and fonts can significantly speed up headless browsing.
This interception approach combines the benefits of both worlds: you use the browser to handle authentication, session management, and JavaScript execution, but you capture the structured API data instead of parsing rendered HTML. It is often the most efficient strategy for scraping modern single-page applications.
Step 6: Handle Infinite Scroll and Lazy Loading
Many modern sites use infinite scroll instead of traditional pagination. Content loads as the user scrolls down the page, with new items appearing dynamically. Scraping these pages requires automating the scroll action and waiting for new content to appear.
The basic pattern is a loop that scrolls to the bottom of the page, waits for new content to load, and checks whether new items appeared. Use Playwright's page.evaluate("window.scrollTo(0, document.body.scrollHeight)") to scroll to the bottom, then page.wait_for_timeout(2000) to give the page time to load new content. Count the number of items before and after scrolling. If no new items appeared, you have reached the end.
For lazy-loaded images and content that only appears when scrolled into view, you may need to scroll progressively rather than jumping to the bottom. Scroll in increments of the viewport height, pausing between each scroll to let content load. This mimics human scrolling behavior and ensures that lazy-loaded elements near the middle of the page are triggered.
Set a maximum scroll count or item count to prevent your scraper from running indefinitely on pages with truly infinite content (like social media feeds). Log the number of items collected after each scroll cycle so you can monitor progress and identify when the page stops returning new content.
An alternative approach for infinite scroll pages is to intercept the API calls that load new batches of content (as described in Step 5). If the page loads data from a paginated API endpoint, you can call that endpoint directly with incrementing offset or page parameters, bypassing the scroll interaction entirely.
Always check for hidden API endpoints before launching a browser. When browser automation is necessary, use Playwright with explicit wait conditions and consider intercepting network responses for cleaner data extraction.