Browser Automation with Python

Step 1: Set Up Python and Install Playwright

You need Python 3.8 or newer installed on your system. Check your Python version by running python --version or python3 --version in your terminal. If you do not have Python installed, download it from python.org and follow the installation instructions for your operating system.

Create a project directory and set up a virtual environment to keep your dependencies isolated. In your terminal, run mkdir browser-automation && cd browser-automation to create the directory, then python -m venv venv to create the virtual environment. Activate it with source venv/bin/activate on macOS and Linux or venv\Scripts\activate on Windows.

Install Playwright with pip: pip install playwright. After pip finishes, run playwright install to download the browser binaries that Playwright needs. This command downloads Chromium, Firefox, and WebKit, which takes a few hundred megabytes of disk space. If you only need one browser, you can specify it: playwright install chromium.

You can verify the installation by running python -c "from playwright.sync_api import sync_playwright; print('Playwright installed successfully')" in your terminal. If you see the success message, the setup is complete.

Step 2: Write Your First Automation Script

Create a new file called automate.py in your project directory. The minimal Playwright script launches a browser, creates a new page, navigates to a URL, and reads the page title. The structure follows a consistent pattern: create a Playwright instance, launch a browser, open a page, perform actions, and close the browser.

The synchronous API wraps everything in a with sync_playwright() as p: context manager. Inside this block, call p.chromium.launch(headless=False) to start a visible Chromium browser. Set headless to True when you do not need to see the browser window. Create a page with browser.new_page(), navigate with page.goto("https://example.com"), and read the title with page.title().

When the script finishes its work, call browser.close() to shut down the browser cleanly. Failing to close the browser leaves orphaned processes that consume system memory. Using a try-finally block or the context manager pattern ensures cleanup happens even when errors occur.

Run your script with python automate.py. You should see a browser window open, navigate to example.com, and then close. The terminal should print the page title. If the browser flashes and closes too quickly to see, add page.wait_for_timeout(3000) before closing to pause for three seconds.

Step 3: Interact with Page Elements

Playwright provides multiple ways to locate elements on a page. The most reliable selectors use element roles, text content, and test IDs rather than CSS class names or DOM positions that can change between deployments.

To click a button, use page.click("text=Submit") to find and click a button by its visible text. Alternatively, use CSS selectors like page.click("#submit-button") for elements with known IDs, or page.click("button.primary") for elements with specific class names. Playwright's built-in locator methods like page.get_by_role("button", name="Submit") are the most resilient option because they match how users perceive the page.

To fill a text input, use page.fill("#email", "user@example.com"). This clears any existing text in the field and types the new value. For password fields, the same method works: page.fill("#password", "your-password"). To select a value from a dropdown, use page.select_option("select#country", "US").

Playwright auto-waits for elements before interacting with them. When you call page.click(), Playwright waits for the element to be visible, enabled, and stable, meaning it is not being animated or repositioned, before performing the click. This eliminates most of the explicit wait logic that older frameworks require. If an element does not appear within the default timeout of 30 seconds, Playwright raises a timeout error with a clear message identifying which element was not found.

For keyboard input, use page.keyboard.press("Enter") to press a specific key or page.keyboard.type("some text") to type a string character by character with realistic timing. For mouse operations, page.mouse.click(x, y) clicks at specific coordinates, which can be useful when dealing with canvas elements or custom UI components that do not respond to standard DOM selectors.

Step 4: Extract Data from Pages

Reading data from the page is one of the most common browser automation tasks. Playwright provides methods to extract text, attributes, and HTML from individual elements or collections of matching elements.

To read the text content of a single element, use page.text_content("h1"). This returns the visible text inside the first h1 element on the page. For the inner text, which strips whitespace and hidden content, use page.inner_text("h1"). To read an HTML attribute, use page.get_attribute("a.product-link", "href").

To extract data from multiple elements, use page.query_selector_all(".product-card") to get a list of all matching elements. Then loop through the list and extract data from each one. A common pattern is to find all product cards on a page and extract the name, price, and URL from each card into a structured data format like a list of dictionaries.

For more complex extraction, use page.evaluate() to run JavaScript directly in the page context. This gives you access to the full browser DOM API. You can execute JavaScript that collects data from the page and returns it to your Python script as a dictionary or list. This approach is powerful for pages with complex data structures that would be tedious to extract element by element.

When scraping multiple pages, combine navigation and extraction in a loop. Navigate to a page, wait for content to load, extract the data, then navigate to the next page. For paginated results, find the "next page" button and click it after each extraction cycle. Always add reasonable delays between page loads to avoid overwhelming the target server.

Step 5: Handle Forms and Authentication

Many automation tasks require logging into a website before performing other actions. Playwright makes this straightforward with its form filling and navigation capabilities.

A basic login flow goes like this: navigate to the login page, fill in the username and password fields, click the submit button, and wait for the authenticated page to load. Use page.goto("https://example.com/login") to reach the login page, page.fill("#username", "your-user") and page.fill("#password", "your-pass") to enter credentials, and page.click("button[type=submit]") to submit the form.

After submitting, wait for navigation to complete with page.wait_for_url("**/dashboard") or page.wait_for_selector(".welcome-message") to confirm that the login succeeded. If the login fails, these waits will time out, which you can catch and handle as an authentication error.

To save login state across sessions, use Playwright's storage state feature. After logging in, call context.storage_state(path="auth.json") to save all cookies and local storage to a file. On subsequent runs, create a new browser context with browser.new_context(storage_state="auth.json") to skip the login process entirely. This saves time on repeated runs and reduces the load on the target server.

For multi-factor authentication, you may need to wait for the user to complete the MFA step manually during the first run, save the session state, and then reuse that state for subsequent automated runs until the session expires.

Step 6: Configure Headless Mode for Production

When you deploy your automation script to a server, CI/CD pipeline, or scheduled task, switch to headless mode. Change p.chromium.launch(headless=False) to p.chromium.launch(headless=True) or simply p.chromium.launch() since headless is the default.

Add error handling for production reliability. Wrap your main automation logic in a try-except block that catches Playwright's TimeoutError and other exceptions. Log meaningful error messages that identify which step failed and include enough context to diagnose the problem without needing to reproduce it manually.

For long-running scraping jobs, implement retry logic for individual page loads. If a page fails to load or an element is not found, log the error and move on to the next page rather than crashing the entire job. Keep track of failed pages so you can retry them in a follow-up run.

Consider resource management in production environments. Each browser instance uses significant memory, typically 100 to 300 megabytes per page depending on the site. Close pages and browsers promptly when they are no longer needed. For high-volume work, reuse browser contexts rather than launching new browser instances for each task.

To run Playwright on a headless Linux server, install the system dependencies with playwright install-deps. This installs the shared libraries that the browser binaries need. For Docker deployments, the official Playwright Docker images include everything pre-configured, which is the simplest path to containerized browser automation.