Selenium with Python

Step 1: Set Up Your Python Environment

Start by ensuring you have Python 3.9 or later installed. Run python --version or python3 --version in your terminal to check. If you need to install Python, download it from python.org and make sure to check "Add Python to PATH" during installation on Windows.

Create a project directory and set up a virtual environment to isolate your dependencies. Run python -m venv venv to create the environment, then activate it with source venv/bin/activate on macOS/Linux or venv\Scripts\activate on Windows. Virtual environments prevent version conflicts between projects and are a Python best practice.

Install Selenium with pip install selenium. This installs the Selenium client library and Selenium Manager, which handles browser driver downloads automatically. For testing, also install pytest with pip install pytest. Your requirements can be saved with pip freeze > requirements.txt for reproducibility.

Verify the installation by opening a Python shell and running from selenium import webdriver. If no errors appear, Selenium is installed correctly. You can also verify by creating a minimal script that opens and closes a browser, confirming that Selenium Manager can find and download the correct driver for your installed browser.

Step 2: Launch and Configure Browsers

Creating a browser instance in Selenium Python starts with importing webdriver and calling the constructor for your target browser. The simplest launch is driver = webdriver.Chrome(), which starts Chrome with default settings and automatic driver management.

Browser behavior is customized through Options objects. For Chrome, create a ChromeOptions instance and add arguments before passing it to the constructor. Common options include --headless=new for headless execution, --disable-gpu for compatibility on some systems, --window-size=1920,1080 to set the viewport size, and --disable-extensions to speed up startup.

Firefox configuration follows the same pattern with FirefoxOptions. Firefox's headless mode is enabled with options.add_argument("-headless"). Firefox also supports profile management through FirefoxProfile, which lets you configure browser preferences like download directories, proxy settings, and content handling rules.

For automated downloads, configure the browser's download behavior through preferences. In Chrome, use options.add_experimental_option("prefs", {"download.default_directory": "/path/to/downloads"}). In Firefox, set preferences on the profile object to control download location and automatic saving of specific MIME types. These configurations are important for scraping workflows that need to download files.

The Service class gives you control over the browser driver process itself. You can specify a custom driver path, set the port it listens on, and configure logging. In most cases, the defaults with Selenium Manager are sufficient, but enterprise environments sometimes require pointing to a specific driver binary managed by their infrastructure.

Always use context managers or try/finally blocks to ensure the browser closes properly. The Pythonic approach is to use driver.quit() in a finally block, which closes all browser windows and terminates the driver process. Unclosed browsers accumulate as zombie processes that consume memory and can eventually crash your machine during long automation runs.

Step 3: Navigate and Interact with Pages

Navigation in Selenium Python uses driver.get(url) to load a page. This method blocks until the page's onload event fires, meaning the HTML document is loaded but dynamically rendered content may still be loading. Use driver.back(), driver.forward(), and driver.refresh() for browser history navigation.

Finding elements uses driver.find_element(By.STRATEGY, value) where the strategy is imported from selenium.webdriver.common.by. The By class provides constants: By.ID, By.NAME, By.CLASS_NAME, By.TAG_NAME, By.CSS_SELECTOR, By.XPATH, By.LINK_TEXT, and By.PARTIAL_LINK_TEXT. Use find_elements() (plural) to get a list of all matching elements.

Clicking an element is straightforward: element.click(). Typing text into input fields uses element.send_keys("text"). To clear an input before typing, call element.clear() first. Reading the visible text of an element uses the .text property, and reading attributes uses element.get_attribute("attribute_name").

The ActionChains class handles complex interactions that go beyond simple clicks and typing. It supports hover (move_to_element), drag and drop, right-click (context_click), double-click, and keyboard combinations. Build a chain of actions with method chaining, then call .perform() to execute them all at once. This is essential for interacting with menus that appear on hover, custom drag interfaces, and keyboard-driven workflows.

JavaScript execution is available through driver.execute_script("javascript code"). This runs JavaScript in the browser context and can return values to your Python code. Common uses include scrolling (window.scrollTo(0, document.body.scrollHeight)), clicking elements hidden behind overlays, reading values from JavaScript variables, and triggering events that Selenium's native methods cannot reach.

Step 4: Handle Forms, Dropdowns, and Alerts

Forms are the most common interaction target in browser automation. A typical form workflow involves finding each input field, clearing any existing values, typing new values with send_keys, and clicking the submit button. For multi-step forms, ensure each step is fully loaded before interacting with the next set of fields.

HTML <select> dropdowns require the Select class from selenium.webdriver.support.ui. Create a Select instance by passing the dropdown element: select = Select(driver.find_element(By.ID, "country")). Then choose an option with select.select_by_visible_text("United States"), select.select_by_value("us"), or select.select_by_index(0). The Select class also provides .options and .all_selected_options properties for reading the available and currently selected choices.

Custom dropdown menus built with divs and JavaScript (not native HTML select elements) cannot use the Select class. Instead, click the dropdown trigger to open the menu, wait for the options to appear, and click the desired option directly. This pattern is common in modern web applications that use component libraries like Material UI, Ant Design, or custom-built select components.

JavaScript alerts, confirms, and prompts require switching to the alert context with alert = driver.switch_to.alert. Read the alert text with alert.text, accept it with alert.accept(), dismiss it with alert.dismiss(), or type into a prompt with alert.send_keys("text") before accepting. Always handle alerts promptly because an unhandled alert blocks all other WebDriver commands.

File upload fields (<input type="file">) are handled by sending the file path directly to the input element: element.send_keys("/path/to/file.pdf"). This bypasses the OS file dialog entirely. For drag-and-drop file uploads that do not use a standard file input, you may need to use JavaScript to construct and dispatch a drop event programmatically.

Checkboxes and radio buttons are clicked like regular elements. Check the current state with element.is_selected() before clicking to avoid toggling a checkbox that is already in the desired state. Radio buttons within the same group are found by name attribute and selected by clicking the specific option you want.

Step 5: Use Explicit Waits and Expected Conditions

Reliable Selenium scripts use explicit waits to synchronize with page loading. Import WebDriverWait from selenium.webdriver.support.ui and expected_conditions as EC from selenium.webdriver.support. These two classes form the foundation of proper wait handling in Python Selenium.

The basic pattern is element = WebDriverWait(driver, timeout).until(EC.condition((By.STRATEGY, value))). The timeout is in seconds. The wait polls the condition repeatedly until it returns a truthy value or the timeout expires, at which point it raises a TimeoutException.

The most frequently used expected conditions are EC.presence_of_element_located (element exists in DOM, may not be visible), EC.visibility_of_element_located (element is visible on the page), EC.element_to_be_clickable (element is visible and enabled), EC.text_to_be_present_in_element (element contains expected text), and EC.staleness_of (an element is no longer attached to the DOM, useful for detecting page transitions).

For conditions not covered by the built-in expected conditions, write a custom condition as a callable that accepts the driver as an argument. Return a truthy value when the condition is met, or False to keep waiting. For example, a custom wait for a specific number of elements: WebDriverWait(driver, 10).until(lambda d: len(d.find_elements(By.CSS_SELECTOR, ".result")) >= 5).

Combine waits with try/except blocks to handle scenarios where an element might or might not appear. For example, a cookie consent banner that only shows on the first visit: try to find and close it with a short timeout, and catch the TimeoutException to continue if it does not appear. This pattern prevents optional elements from failing your scripts.

Set a reasonable default timeout based on your application's behavior. Most web applications load critical content within 5-10 seconds, so a 10-second timeout is a sensible default. Increase the timeout for pages with heavy data loading or external API dependencies, but avoid extremely long timeouts that hide genuine performance problems.

Step 6: Structure a Real Automation Project

Small scripts work fine as single files, but real automation projects need structure. The Page Object Model (POM) is the standard architectural pattern for Selenium projects of any language, and Python's class system makes it clean to implement.

Each page or major component of the application gets its own class. The class defines locators as class attributes and interaction methods that use those locators. For example, a LoginPage class would define USERNAME_INPUT, PASSWORD_INPUT, and LOGIN_BUTTON locators, plus a login(username, password) method that fills the fields and clicks the button. Test code calls login_page.login("user", "pass") instead of manipulating elements directly.

This separation means that when the application's UI changes, you update one page object class instead of every test that touches that page. It also makes tests more readable because the test steps read like user actions ("login," "search for product," "add to cart") rather than low-level element operations.

For pytest integration, define your WebDriver as a fixture in conftest.py. A session-scoped fixture creates one browser for the entire test session, while a function-scoped fixture creates a fresh browser for each test. Function scope provides better isolation but is slower. A balanced approach uses session scope for the browser and creates fresh sessions or navigates to a clean state between tests.

Configuration management keeps environment-specific values (base URL, credentials, browser choice, timeout values) separate from test code. Use a simple Python config file, environment variables, or a library like python-dotenv. Never hardcode URLs or credentials in test files because they change between development, staging, and production environments.

A well-structured Selenium Python project has directories for page objects, test files, configuration, and utilities (custom waits, data generators, reporting helpers). This organization scales from small projects with a handful of tests to large suites with hundreds of test cases across multiple applications.