Headless Browser Automation in Python

Step 1: Install the Library and Browser

For Playwright, installation is a two-step process. First, install the Python package with pip install playwright. Then download the browser binaries with playwright install. This command downloads Chromium, Firefox, and WebKit, each about 100-200 MB. To download only Chromium, use playwright install chromium. On Linux servers, add playwright install --with-deps to also install required system libraries like libgbm, libatk, and libnss3.

For Selenium, install the package with pip install selenium. Selenium 4+ includes Selenium Manager, which automatically downloads ChromeDriver when you first create a Chrome WebDriver instance. If your system already has Chrome installed, Selenium Manager detects its version and downloads the matching ChromeDriver. For environments without Chrome, you will need to install Chrome separately through your system's package manager or by downloading it from Google.

Both libraries work with Python 3.8 and above. Playwright also offers an async API built on asyncio, which is valuable for applications that need to run multiple browser sessions concurrently without threading. This async support integrates naturally with Python web frameworks like FastAPI and aiohttp.

Step 2: Write a Basic Headless Script

A minimal Playwright script in Python uses the synchronous API and context manager pattern. You start Playwright, launch a browser, create a page, navigate to a URL, interact with the page, and close everything. The with sync_playwright() as p: context manager handles startup and cleanup automatically, ensuring the browser process is terminated even if your script encounters an error.

The core pattern is: browser = p.chromium.launch() to start headless Chromium, page = browser.new_page() to create a tab, page.goto('https://example.com') to navigate, and then use methods like page.title(), page.content(), or page.query_selector() to read page data. Call browser.close() when finished. The browser launches in headless mode by default, with no extra configuration required.

A minimal Selenium script follows a similar flow but with different syntax. Create options with options = webdriver.ChromeOptions(), add options.add_argument('--headless=new'), instantiate driver = webdriver.Chrome(options=options), navigate with driver.get('https://example.com'), and clean up with driver.quit(). The driver object provides methods like find_element(), page_source, and title for page interaction.

Step 3: Handle Dynamic Content and JavaScript

Modern websites load content asynchronously through JavaScript. When you navigate to a page, the initial HTML may not contain the data you need because it is loaded by scripts after the page renders. Handling this correctly is one of the main reasons to use a headless browser instead of an HTTP library like Requests.

Playwright handles this elegantly with auto-waiting. When you call page.click('button'), Playwright automatically waits for the button to appear in the DOM, become visible, become enabled, and be stable (not animating) before clicking. This eliminates the most common class of automation failures. For explicit waits, page.wait_for_selector('.result') pauses execution until the specified element appears, and page.wait_for_load_state('networkidle') waits until no network requests have been made for 500 milliseconds.

Selenium requires more explicit wait configuration. The WebDriverWait class combined with expected_conditions provides explicit waits: WebDriverWait(driver, 10).until(EC.presence_of_element_located((By.CSS_SELECTOR, '.result'))). Without explicit waits, Selenium will attempt to find elements immediately after navigation, often failing because the JavaScript has not yet rendered the content. Setting an implicit wait with driver.implicitly_wait(10) adds a global timeout for element finding, but explicit waits give you finer control over which conditions to wait for.

Step 4: Extract Data from Pages

Data extraction in Playwright uses CSS selectors or XPath expressions to locate elements, then methods to read their properties. Use page.query_selector('.price') to find a single element, page.query_selector_all('.product') to find all matching elements, and element.text_content() to read the text inside an element. For attributes, use element.get_attribute('href'). To read the entire HTML of an element, use element.inner_html().

Playwright also supports evaluating JavaScript directly in the page context with page.evaluate(). This lets you run arbitrary JavaScript and return the result to your Python code: result = page.evaluate('document.querySelectorAll(".item").length'). For complex extraction logic, writing the extraction in JavaScript and returning structured data is often cleaner than navigating the DOM through Playwright's Python API.

Selenium uses similar patterns with slightly different method names. driver.find_element(By.CSS_SELECTOR, '.price') finds a single element, driver.find_elements(By.CSS_SELECTOR, '.product') finds all matches. Read text with the .text property and attributes with .get_attribute('href'). Selenium also supports JavaScript execution through driver.execute_script().

For structured data extraction, both libraries support iterating over collections of elements to build lists or dictionaries. A common pattern is to find all product containers, then within each container, extract the title, price, image URL, and link into a dictionary, building a list of product records that can be saved to JSON, CSV, or a database.

Step 5: Apply Production Best Practices

Production headless browser scripts need error handling, resource management, and environment-specific configuration. Always use try/finally or context managers to ensure the browser is closed even if an error occurs. Leaked browser processes consume memory indefinitely and can crash the host system in long-running applications.

For Playwright, the with sync_playwright() as p: context manager handles cleanup. For scripts that create multiple browser contexts, close each context when finished. Set browser.new_context(viewport={'width': 1920, 'height': 1080}) to control viewport size, and use context.set_default_timeout(30000) to prevent scripts from hanging on unresponsive pages.

For Selenium, always call driver.quit() (not just driver.close()) to terminate the browser process. Add the --disable-dev-shm-usage flag in Docker environments where the /dev/shm shared memory partition is too small. Set page load timeouts with driver.set_page_load_timeout(30) to prevent navigation from blocking indefinitely on slow-loading pages.

Logging is essential for debugging production automation. Log each navigation, the current URL, any errors encountered, and the data extracted. When a script fails at 3 AM on a weekend, good logs are the difference between a quick diagnosis and hours of guesswork. Both Playwright and Selenium provide access to browser console messages that can be captured in your application logs.

For concurrent automation, Playwright's async API with asyncio allows multiple browser pages or contexts to run in a single Python process without the overhead of threading. Selenium requires threading or multiprocessing for concurrency, which adds complexity around thread safety and resource sharing. In either case, monitor memory usage carefully, as each browser page typically uses 100-200 MB of RAM depending on page complexity.