Playwright with Python

Step 1: Install Playwright for Python

Playwright for Python requires Python 3.8 or later. The installation is a two-step process: first install the Python package, then download the browser binaries that Playwright controls.

Install the package with pip: pip install playwright. This installs the Playwright library and its dependencies. For test projects, install the pytest plugin as well: pip install pytest-playwright.

Next, download the browser binaries: playwright install. This downloads Chromium, Firefox, and WebKit builds specifically compiled for Playwright. If you only need one browser, specify it: playwright install chromium. On Linux servers or CI environments, add --with-deps to automatically install the system libraries that browsers require.

Verify the installation by running playwright --version to confirm the package is accessible, and python -c "from playwright.sync_api import sync_playwright; print('OK')" to verify the Python bindings load correctly.

For project isolation, use a virtual environment. Create one with python -m venv venv, activate it, and then install Playwright inside it. This keeps Playwright's dependencies separate from your system Python packages and makes your project reproducible across machines.

Step 2: Choose Between Sync and Async APIs

Playwright for Python provides two complete API surfaces: synchronous and asynchronous. Both offer identical functionality, so the choice depends on your project's architecture and personal preference.

The synchronous API (from playwright.sync_api import sync_playwright) is straightforward and reads like sequential code. Each operation blocks until complete, so page.goto(url) does not return until the page has loaded. This API is ideal for scripts, simple automation tasks, and test suites where you do not need concurrent operations.

The asynchronous API (from playwright.async_api import async_playwright) uses Python's asyncio framework. Each operation returns a coroutine that you await, which allows you to run multiple browser operations concurrently. This API is better suited for scraping applications that need to process multiple pages simultaneously, or for integration into existing async codebases built on frameworks like FastAPI or aiohttp.

The sync API is recommended for beginners and most testing scenarios. The mental model is simpler since each line executes in order, and debugging is more straightforward because the call stack is linear. Switch to the async API only when you have a concrete need for concurrency, such as scraping hundreds of pages in parallel or integrating with an async web server.

One important constraint: the synchronous API cannot be used inside an already-running async event loop. If your code runs within an async framework (like a Jupyter notebook or an async web server), you must use the async API. Attempting to use the sync API in these contexts raises an error.

Step 3: Write a Basic Automation Script

A basic Playwright Python script follows a consistent pattern: start Playwright, launch a browser, create a page, perform actions, and clean up. The sync API uses a context manager to handle resource cleanup automatically.

The sync_playwright() context manager starts the Playwright server process. Within it, you call playwright.chromium.launch() to start a Chromium browser (or .firefox or .webkit for those engines). The headless parameter controls whether the browser window is visible, which defaults to True for scripts and CI environments.

Once you have a browser instance, create a new page with browser.new_page(). The page object is your primary interface for automation. Use page.goto(url) to navigate, page.locator(selector) to find elements, .click() to click, .fill(value) to type into inputs, and .text_content() or .inner_text() to extract text.

For example, a script that navigates to a search engine, enters a query, and captures the results page would call page.goto() for navigation, page.get_by_role("searchbox").fill("query") for typing, page.keyboard.press("Enter") to submit, page.wait_for_load_state("networkidle") to ensure results have loaded, and page.screenshot(path="results.png") to capture the output.

Always close the browser when finished, either explicitly with browser.close() or by using the context manager pattern which handles this automatically. Failing to close the browser leaves orphaned processes consuming system resources.

Step 4: Set Up Pytest Integration

The pytest-playwright plugin provides the most productive workflow for writing Playwright tests in Python. It manages browser lifecycle, provides fixtures for common objects, and integrates with pytest's reporting and assertion system.

After installing pytest-playwright, you get automatic access to several fixtures. The page fixture provides a fresh browser page for each test. The context fixture gives you the browser context, and the browser fixture gives you the browser instance. Most tests only need the page fixture.

A test function that receives the page fixture can immediately start interacting with the browser. There is no setup or teardown code needed, as the fixture handles launching the browser before the test and closing it after. Each test gets its own isolated browser context, so cookies, storage, and other state do not leak between tests.

Assertions use the expect function from playwright.sync_api. Calling expect(page).to_have_title("Expected Title") waits for the title to match, retrying automatically until the timeout expires. These web-first assertions are essential for reliable tests because web pages often update asynchronously after navigation or interaction.

Configuration goes in pytest.ini, pyproject.toml, or conftest.py. You can set the base URL, default browser, viewport size, and whether to capture screenshots or traces on failure. The --browser command-line flag lets you run tests in a specific browser: pytest --browser chromium or pytest --browser firefox.

Run tests with pytest as usual, with Playwright-specific options available: pytest --headed to see the browser, pytest --slowmo 500 to add delays between actions, and pytest --tracing on to capture traces for every test.

Step 5: Handle Authentication and State

Most web applications require authentication, and logging in before every test or scraping session wastes time. Playwright's storage state feature lets you authenticate once and reuse that session across all subsequent operations.

The approach is straightforward: run a login flow in one script or setup step, then save the browser context's storage state to a JSON file using context.storage_state(path="auth.json"). This file contains all cookies and local storage entries from the authenticated session.

In subsequent scripts or tests, create a new browser context with the saved state: browser.new_context(storage_state="auth.json"). The new context starts with all the cookies and storage from the saved session, so you are immediately logged in without going through the login flow again.

For pytest integration, create a fixture in conftest.py that runs the login flow once per session and saves the state. Other test functions load this state when creating their browser context. The pytest-playwright plugin supports this pattern through the browser_context_args fixture, where you can specify the storage state path.

Keep the authentication state file out of version control since it contains session tokens. Add it to .gitignore and regenerate it in CI by running the login setup as a prerequisite step. For applications with short-lived sessions, you may need to regenerate the state periodically or handle re-authentication gracefully in your scripts.

Step 6: Build a Web Scraping Script

Playwright excels at scraping websites that load content dynamically with JavaScript. Unlike HTTP-based scrapers like requests or Scrapy, Playwright renders the full page in a real browser, so all JavaScript-generated content is accessible in the DOM.

A scraping script starts the same way as any automation script: launch a browser, create a page, navigate to the target URL. The key difference is in how you extract data. Use page.query_selector_all(selector) to find all matching elements, then iterate through them to extract text, attributes, or structured data.

For pages with infinite scroll or lazy loading, you need to scroll the page to trigger content loading. Use page.evaluate("window.scrollTo(0, document.body.scrollHeight)") followed by a wait for new content to appear. Repeat until no new content loads or you have collected enough data.

Network interception is a powerful scraping technique with Playwright. Many modern websites load data through API calls, and intercepting these requests with page.route() or page.on("response") lets you capture structured JSON data directly instead of parsing HTML. This is faster, more reliable, and gives you cleaner data than DOM scraping.

Performance optimization for scraping includes blocking unnecessary resources (images, fonts, CSS) with route interception, using multiple browser contexts for parallel scraping, and running in headless mode to reduce resource consumption. For large-scale scraping, consider using the async API to process multiple pages concurrently within a single script.