Web Scraping with Playwright

Step 1: Set Up Playwright for Scraping

The setup for scraping is similar to testing but with different configuration priorities. Install Playwright for your language of choice (JavaScript or Python are most common for scraping) and download the browser you plan to use. Chromium is the default choice because it has the largest market share and is least likely to trigger anti-bot measures that target unusual browser engines.

Configure the browser launch for scraping efficiency. Run in headless mode (headless: true) to eliminate the overhead of rendering a visible window. Set the viewport to a standard desktop resolution like 1920x1080 to ensure pages render their desktop layout, which typically contains more data than mobile views.

Create a browser context with realistic settings. Set a common user agent string, enable JavaScript, and configure the locale and timezone to match the target site's expected audience. These settings reduce the chance of being identified as an automated browser. Playwright's browser contexts are lightweight, so creating new ones for different scraping tasks adds negligible overhead.

For scraping behind authentication, log in once and save the storage state with context.storageState({ path: 'auth.json' }). Subsequent scraping sessions load this state to skip the login flow: browser.newContext({ storageState: 'auth.json' }). Regenerate the state when sessions expire.

Step 2: Extract Data from Dynamic Pages

The core of any scraping task is extracting data from the rendered page. Playwright provides several approaches, each suited to different content types and page structures.

For text content, use locators to find elements and extract their text. page.locator('.product-name').allTextContents() returns an array of text from all matching elements. For individual elements, .textContent() returns the raw text including whitespace, while .innerText() returns the visible text with whitespace normalized.

For attributes, use .getAttribute('href') to extract link URLs, .getAttribute('src') for image sources, or .getAttribute('data-price') for custom data attributes. These methods work on individual locator results, so combine them with .all() to process multiple elements in a loop.

For complex data extraction, use page.evaluate() to run JavaScript directly in the browser context. This method accepts a function that executes in the page's JavaScript environment, with full access to the DOM and any JavaScript variables or APIs the page uses. The return value is serialized and sent back to your script. This is the most flexible extraction method and is useful when the data you need is stored in JavaScript objects rather than visible DOM elements.

Wait for content to be present before extracting it. Dynamic pages often load content after the initial page load through API calls or lazy rendering. Use page.waitForSelector('.data-container') to wait for a specific element, page.waitForLoadState('networkidle') to wait until network activity settles, or page.waitForResponse(url) to wait for a specific API response that populates the page.

Step 3: Intercept Network Requests

Many modern websites load data through internal API endpoints, rendering the response with JavaScript. Intercepting these API responses is often more efficient than parsing the rendered HTML, because the API returns structured JSON data that is cleaner and more complete than what appears in the DOM.

Use page.on('response') to listen for network responses. Filter by URL pattern to capture only the API calls you care about, then call response.json() to parse the response body. This gives you the exact data the page uses to render its content, without the overhead of DOM parsing.

For pages that load data through multiple API calls, set up listeners before navigating to the page. The listener captures responses as they arrive, and you can accumulate data from multiple requests. This is common on sites with infinite scroll where each scroll triggers a new API call for the next batch of results.

You can also use page.route() to modify or block requests. Block tracking scripts, analytics, and ads to speed up page loads. Modify request headers to add authentication tokens or change the User-Agent. Fulfill requests with mock data when testing your scraping logic against predictable inputs.

A powerful pattern combines navigation with response interception: navigate to the page, wait for the specific API response you need, extract the JSON data from that response, and skip DOM parsing entirely. This approach is faster, more reliable, and produces cleaner data than HTML scraping for sites that use API-driven rendering.

Step 4: Handle Pagination and Infinite Scroll

Most data sources spread their content across multiple pages or use infinite scroll to load content incrementally. Handling pagination correctly is essential for comprehensive data extraction.

For traditional pagination with numbered page links or next buttons, the approach is straightforward. After extracting data from the current page, locate the next page link and click it, wait for the new content to load, and repeat. Check for the absence of the next link to know when you have reached the last page. Track visited URLs to avoid cycles on sites with non-linear pagination.

For infinite scroll pages, automate the scrolling behavior that triggers content loading. Use page.evaluate(() => window.scrollTo(0, document.body.scrollHeight)) to scroll to the bottom, then wait for new content to appear. The most reliable approach monitors the number of items on the page: scroll, wait, count items, and stop when the count stops increasing.

For "Load More" buttons, locate the button and click it after each batch of content loads. Wait for the new content to render before clicking again. Stop when the button disappears or becomes disabled, which indicates all content has been loaded.

Rate limiting is important for any pagination strategy. Adding delays between page loads reduces the load on the target server and decreases the likelihood of being blocked. Use page.waitForTimeout() or your language's sleep function to add a randomized delay between 1 and 3 seconds between requests, mimicking natural browsing behavior.

Step 5: Block Unnecessary Resources

Scraping does not need images, fonts, CSS, or tracking scripts. Blocking these resources reduces bandwidth consumption, speeds up page loads, and decreases the memory footprint of your scraper. On media-heavy pages, blocking images alone can reduce load time by 50% or more.

Use page.route() to intercept and abort requests by resource type. The request object provides a resourceType() method that returns values like 'image', 'stylesheet', 'font', 'media', and 'script'. Abort requests for types you do not need by calling route.abort().

Be selective about which scripts to block. Blocking all scripts will prevent JavaScript from running, which defeats the purpose of using a browser-based scraper. Instead, block specific domains known for analytics and tracking (Google Analytics, Facebook Pixel, advertising networks) while allowing the site's own scripts to execute. URL pattern matching in page.route() supports glob patterns, making it easy to target specific domains.

Another optimization is to disable CSS rendering by blocking stylesheets. Since you are extracting data rather than viewing the page, visual styling is unnecessary. This can significantly speed up rendering on CSS-heavy pages. However, if your selectors depend on computed styles or visible text (which can be affected by CSS display properties), keep stylesheets enabled.

For pages with video or audio content, block the 'media' resource type to prevent large media files from downloading. These files are the biggest bandwidth consumers and provide no value for data extraction tasks.

Step 6: Scale with Multiple Contexts

When you need to scrape many pages, running them sequentially is too slow. Playwright's browser context architecture lets you run multiple scraping sessions in parallel within a single browser instance, sharing the browser process's resources efficiently.

Create multiple browser contexts, each acting as an independent session with its own cookies, cache, and state. Assign a subset of URLs to each context and process them concurrently. In JavaScript, use Promise.all() to run context-based scrapers in parallel. In Python, use the async API with asyncio.gather().

The number of parallel contexts depends on your machine's resources. Each context consumes memory proportional to the pages it renders. A reasonable starting point is 5-10 parallel contexts on a machine with 8GB of RAM. Monitor memory usage and adjust the concurrency level based on your observations. Pages with heavy JavaScript or large DOM trees consume more memory than simple pages.

For truly large-scale scraping (thousands or millions of pages), distribute the work across multiple machines. Each machine runs its own Playwright instance with multiple contexts. A task queue like Redis or RabbitMQ coordinates work assignment, and results are stored in a shared database or file store. This architecture scales horizontally while keeping each individual scraper simple.

Error handling is critical in parallel scraping. Individual pages may fail due to network timeouts, unexpected content, or anti-bot measures. Wrap each page's scraping logic in a try-catch block, log failures for retry, and continue processing other pages. A failed page should not crash the entire scraping session. Implement exponential backoff for retries to avoid hammering a site that may be rate-limiting you.