How to Scrape Dynamic Pages with JavaScript
A growing percentage of the web is built with JavaScript frameworks like React, Vue, Angular, and Svelte. These frameworks render content on the client side, meaning the HTML that the server returns contains little or no visible data. The actual product listings, search results, user profiles, and other content only appear after the browser executes JavaScript, makes API calls to backend services, and inserts the resulting data into the DOM. If you try to scrape these pages with Cheerio or any other static HTML parser, you get an empty shell. Browser automation solves this by running a real browser that processes the page exactly like a human visitor's browser would.
Step 1: Identify Dynamic Content
Before reaching for browser automation, confirm that the target page actually requires it. Many developers assume a page is dynamic when it is not, wasting resources on unnecessary browser instances. There are two reliable ways to check.
The first method is viewing the page source. In Chrome or Firefox, right-click the page and select "View Page Source" (not "Inspect"). This shows the raw HTML that the server returned before any JavaScript executed. Search for the data you want to scrape. If you can find the product names, prices, or other target data in the source, the page is static and you can use Cheerio. If the source contains only empty div containers, script tags, and framework boilerplate, the content is rendered dynamically.
The second method is making a request with curl or Axios and examining the response. Run curl -s "https://target-site.com/page" | head -100 in your terminal, or fetch the page with Axios and log the response. If the response HTML contains your target data, use Cheerio. If it contains a root div with a script bundle and no visible content, you need browser automation.
There is an important third option to check before committing to browser automation. Open your browser's developer tools, switch to the Network tab, and reload the target page. Filter by XHR or Fetch requests. Many dynamic pages fetch their data from REST or GraphQL API endpoints that return structured JSON. If you can identify these endpoints, you can call them directly with Axios and skip the browser entirely. This approach is faster, uses less memory, and produces cleaner data than parsing rendered HTML. Look for requests that return JSON objects containing the data you see on the page.
Step 2: Choose Your Browser Tool
Node.js has two major browser automation libraries: Puppeteer and Playwright. Both launch real browser instances and provide high-level APIs for navigation, interaction, and data extraction. The choice between them depends on your specific requirements.
Puppeteer is Google's official library for controlling Chrome and Chromium. It uses the Chrome DevTools Protocol for browser communication and provides a mature, well-documented API. Puppeteer downloads a compatible version of Chromium automatically when you install it via npm. Install with npm install puppeteer for the full package with Chromium, or npm install puppeteer-core if you want to provide your own browser installation.
Playwright is Microsoft's library that supports three browser engines: Chromium, Firefox, and WebKit (Safari). Its auto-waiting feature automatically waits for elements to be visible, enabled, and stable before interacting with them, which eliminates many timing-related bugs that plague Puppeteer scrapers. Playwright also supports browser contexts (isolated sessions within a single browser) for efficient parallel scraping. Install with npm install playwright, then run npx playwright install to download browser binaries.
For most new scraping projects, Playwright is the better choice. Its auto-waiting reduces flaky behavior, its browser contexts are more memory-efficient for parallel scraping, and its API is slightly more intuitive. Puppeteer remains a solid option if you only need Chrome support, have existing Puppeteer code, or prefer its slightly smaller installation footprint (one browser instead of three).
Step 3: Launch and Navigate
Both Puppeteer and Playwright follow the same three-step pattern: launch a browser, create a page, and navigate to a URL. The browser runs in headless mode by default (no visible window), which is faster and uses less memory than headed mode.
With Playwright, the basic pattern is: call chromium.launch() to start a browser, call browser.newPage() to create a page, then call page.goto(url) to navigate. The goto() method returns a promise that resolves when the page finishes loading. You specify what "finished loading" means with the waitUntil option.
The waitUntil option controls when the navigation is considered complete. The value 'domcontentloaded' resolves when the HTML document has been parsed but external resources (images, stylesheets) may still be loading. The value 'load' resolves when the page has fully loaded including all images and stylesheets. The value 'networkidle' resolves when there have been no network requests for 500 milliseconds, which usually means all AJAX calls have completed. For scraping dynamic content, 'networkidle' is typically the best choice because it waits for JavaScript-initiated data fetches to finish. However, some pages have persistent connections (WebSockets, polling) that prevent networkidle from ever resolving. In those cases, use 'domcontentloaded' combined with explicit element waiting.
Always close the browser when you are done with browser.close(). Use a try/finally block to ensure the browser closes even if your scraping code throws an error. Leaving browser instances running leaks memory and can exhaust system resources.
Step 4: Wait for Dynamic Content
Even after goto() resolves, some content may still be loading. Interactive elements, lazily loaded images, content behind "Load More" buttons, and data from slow API calls may not be present yet. Explicit waiting strategies ensure you extract data only after it has fully appeared in the DOM.
The page.waitForSelector(selector) method pauses execution until an element matching the CSS selector appears in the DOM. This is the most common waiting strategy for scraping. If you know that product cards have the class "product-item", call await page.waitForSelector('.product-item') before extracting data. You can set a timeout to avoid waiting forever: await page.waitForSelector('.product-item', { timeout: 10000 }) waits up to 10 seconds before throwing an error.
The page.waitForFunction(fn) method pauses until a JavaScript function returns a truthy value. This is useful for waiting on conditions that are not easily expressed as CSS selectors. For example, waiting until a specific number of items have loaded: await page.waitForFunction(() => document.querySelectorAll('.product-item').length >= 20). Or waiting until a loading spinner disappears: await page.waitForFunction(() => !document.querySelector('.loading-spinner')).
The page.waitForResponse(urlPattern) method pauses until the browser receives a response matching the URL pattern. This is ideal when you know which API call delivers the data you need. You can wait for the specific API response, then extract data from either the API response body or the rendered DOM.
For pages with infinite scroll, you need to combine scrolling with waiting. Scroll to the bottom of the page with page.evaluate(() => window.scrollTo(0, document.body.scrollHeight)), then wait for new content to appear with page.waitForSelector() or page.waitForFunction(). Repeat this cycle until no new content loads. Track the number of items before and after each scroll to detect when you have reached the end.
Step 5: Extract Data from the Rendered DOM
Once the page is fully loaded and all dynamic content has appeared, you extract data using page.evaluate() or Playwright's locator API. Both approaches give you access to the fully rendered DOM, including all content that JavaScript added after the initial page load.
The page.evaluate(fn) method runs a JavaScript function inside the browser's execution context. Inside this function, you have access to the full browser DOM API: document.querySelector(), document.querySelectorAll(), element.textContent, element.getAttribute(), and everything else. The function's return value is serialized and passed back to your Node.js script. Return plain objects, arrays, strings, and numbers. You cannot return DOM elements or functions because they cannot be serialized across the browser-to-Node boundary.
A typical extraction pattern with page.evaluate() selects all items matching a CSS selector, iterates over them with Array.from() and map(), and extracts text and attributes from each one. The function returns an array of plain objects, each containing the scraped fields for one item. Back in your Node.js code, you receive this array as a regular JavaScript variable ready for processing or storage.
Playwright's locator API provides an alternative that does not require page.evaluate(). Locators like page.locator('.product-item') represent page elements with automatic waiting and retry logic. You can extract text with await locator.textContent(), attributes with await locator.getAttribute('href'), and iterate with await locator.count() and locator.nth(index). The locator API is slightly slower than page.evaluate() because each call crosses the browser-Node boundary, but it is more readable and less error-prone for simple extractions.
For large-scale scraping where performance matters, page.evaluate() is faster because it makes a single round-trip to the browser to extract all data at once. For small-scale scraping where readability matters more than speed, the locator API is cleaner and easier to debug.
Step 6: Intercept API Calls
Network interception is the most powerful and often overlooked technique for scraping dynamic pages. Instead of waiting for the browser to render content and then parsing the HTML, you capture the API responses that contain the raw data. This approach is faster, produces cleaner data, and is less affected by UI changes on the target site.
Both Puppeteer and Playwright support request and response interception. In Playwright, you register a listener with page.on('response', async (response) => { ... }) before navigating to the page. Inside the callback, check the response URL to determine if it matches the API endpoint that delivers your target data. If it does, call await response.json() to get the parsed JSON body, which contains structured data that you can use directly without any HTML parsing.
To identify which API endpoints to intercept, open the target page in your browser's developer tools and switch to the Network tab. Filter by XHR or Fetch requests and look for responses that contain JSON data matching what you see on the page. Note the URL patterns, as they typically include the domain, an API path, and query parameters. In your scraper, match against these URL patterns to capture the right responses.
Network interception also lets you block unnecessary requests to speed up page loading. Images, fonts, CSS files, analytics scripts, and advertisement scripts are not needed for data extraction and consume bandwidth and rendering time. In Playwright, use page.route('**/*.{png,jpg,gif,css,woff2}', route => route.abort()) to block these resources. In Puppeteer, enable request interception with page.setRequestInterception(true) and abort requests by resource type. Blocking non-essential resources can reduce page load times by 50 to 80 percent, which adds up significantly when scraping thousands of pages.
A combined approach often works best: intercept API responses for structured data, but also extract some elements from the rendered DOM for data that is not available through API calls (such as UI labels, navigation structure, or metadata embedded in the HTML). This hybrid strategy gives you the best of both worlds.
Handling Common Dynamic Page Patterns
Several dynamic page patterns require specific scraping approaches beyond the basic steps above.
Single-Page Applications (SPAs) built with React, Angular, or Vue use client-side routing, which means navigating between pages does not trigger a full page reload. When scraping multiple pages of an SPA, do not close and reopen the browser for each page. Instead, use page.click() on navigation links and wait for new content to load. Alternatively, intercept the API calls triggered by navigation and extract data from the JSON responses.
Login-protected content requires authenticating before scraping. Navigate to the login page, fill in credentials using page.fill() or page.type(), click the submit button with page.click(), and wait for the authentication to complete. After login, the browser session maintains cookies and authentication state for subsequent page visits. Save the authentication state using Playwright's context.storageState() to avoid re-logging in on every scraper run.
Modals, popups, and overlays (cookie consent banners, newsletter popups, age gates) can block access to underlying content. Use page.click() to dismiss these elements before extracting data. Check for their presence with a short timeout to avoid errors on pages where they do not appear: try { await page.click('.cookie-accept', { timeout: 3000 }); } catch (e) { /* no cookie banner */ }.
Shadow DOM elements, used by some modern web components, are not visible to regular CSS selectors. Playwright handles shadow DOM automatically with its page.locator() method, which pierces shadow boundaries by default. Puppeteer requires using element.shadowRoot within page.evaluate() to access shadow DOM content.
Always check for API endpoints before scraping dynamic pages with a browser. Direct API calls are faster, cheaper, and produce cleaner data than rendering and parsing HTML. When browser automation is genuinely necessary, use Playwright for its superior auto-waiting and choose the right waiting strategy (waitForSelector, waitForFunction, or response interception) for your specific target page.