Web Scraping with Selenium

Step 1: Set Up Selenium for Scraping

Scraping with Selenium starts with a properly configured headless browser. Headless mode runs the browser without rendering a visible window, which saves system resources and is required for server environments without a display. In Python, configure Chrome for scraping with ChromeOptions:

Set --headless=new to enable Chrome's modern headless mode, which provides full rendering capability without a visible window. Add --disable-gpu for compatibility on Linux systems, --window-size=1920,1080 to ensure the page renders at desktop resolution (some sites serve different content based on viewport size), and --disable-blink-features=AutomationControlled to remove one of the signals that anti-bot systems check.

Block unnecessary resource loading to speed up page loads. Selenium does not have built-in request blocking like Playwright, but you can use Chrome DevTools Protocol (CDP) commands through Selenium 4 to block image, font, and media requests. Alternatively, configure a proxy or browser extension that filters requests. Every blocked resource reduces bandwidth and loading time, which compounds significantly during large scraping runs.

Set a realistic user agent string with --user-agent="Mozilla/5.0 ..." to match a real browser profile. The default Selenium user agent sometimes includes identifiers that anti-bot systems flag. Use a current user agent string from a real Chrome or Firefox installation on the same operating system you are running.

For large-scale scraping, consider running multiple browser instances in parallel. Python's concurrent.futures module with ThreadPoolExecutor or ProcessPoolExecutor can manage parallel Selenium sessions. Each thread or process gets its own WebDriver instance, so sessions are fully isolated. Be mindful of memory usage since each Chrome instance consumes 200-500 MB of RAM depending on page complexity.

Step 2: Extract Data from Dynamic Pages

Dynamic pages load content asynchronously through JavaScript after the initial HTML document arrives. A simple HTTP request to these pages returns an empty shell with JavaScript code that would normally populate the content. Selenium handles this automatically because it runs the full browser engine, including JavaScript execution.

The key to extracting data from dynamic pages is waiting for the content to actually appear before trying to read it. Use explicit waits with WebDriverWait and expected_conditions to pause until the elements you need are present in the DOM or visible on the page. For example, waiting for product listings to load on an e-commerce site: WebDriverWait(driver, 15).until(EC.presence_of_all_elements_located((By.CSS_SELECTOR, ".product-card"))).

Once the content is loaded, extract data using Selenium's element locators. Find elements by CSS selector, XPath, or other strategies, then read their text content with .text and attributes with .get_attribute(). For structured data extraction, find all container elements (like product cards or table rows), then find child elements within each container to build data records.

An alternative extraction approach is to grab the rendered page source with driver.page_source and parse it with BeautifulSoup. This hybrid method uses Selenium for rendering and navigation but delegates HTML parsing to BeautifulSoup, which has more powerful and flexible selection capabilities than Selenium's locators. The pattern is common in Python scraping projects: soup = BeautifulSoup(driver.page_source, "html.parser").

For pages that load data through API calls, you can intercept the underlying requests instead of parsing HTML. Use Selenium 4's CDP integration or browser developer tools to identify the API endpoints the page calls, then extract JSON data directly from those responses. This approach is faster and more reliable than HTML parsing because API responses have consistent structure and do not change when the visual layout is redesigned.

Step 3: Handle Pagination and Infinite Scroll

Most websites that display lists of content use either traditional pagination (numbered page links or next/previous buttons) or infinite scroll (loading more content as you scroll down). Selenium handles both patterns, but each requires a different strategy.

For traditional pagination, locate the "Next" button or page number links and click through them in a loop. After each click, wait for the new content to load before extracting data. A robust pagination loop checks for the existence of the next button before clicking, which naturally stops at the last page. Keep track of which page you are on and implement a maximum page limit to prevent runaway scraping of sites with thousands of pages.

URL-based pagination is simpler and faster. If the page number is visible in the URL (like ?page=2 or /results/page/3), you can construct URLs directly with driver.get() instead of clicking pagination controls. This approach avoids the overhead of finding and clicking elements, and it naturally handles sites where pagination controls are inconsistent or hard to locate.

Infinite scroll pages load new content when the user scrolls near the bottom. Simulate this in Selenium by executing JavaScript to scroll to the bottom of the page: driver.execute_script("window.scrollTo(0, document.body.scrollHeight)"). After scrolling, wait for new content to load, then scroll again. Repeat until no new content appears. Track the page height before and after each scroll to detect when you have reached the end of the content.

Some infinite scroll implementations use "Load More" buttons instead of automatic loading. For these, locate the button, click it, wait for the new content, and repeat. Combine a click loop with a check for the button's existence or visibility to stop when all content has been loaded.

Add delays between page loads to be respectful of the target server. A random delay between 1 and 3 seconds between requests simulates human browsing patterns and reduces the risk of being rate-limited or blocked. Use Python's time.sleep(random.uniform(1, 3)) between pagination steps. This is one of the few legitimate uses of time.sleep in Selenium, since you are deliberately throttling rather than waiting for an element.

Step 4: Scrape Behind Login Walls

Many valuable data sources require authentication before displaying content. Selenium excels at this because it can fill login forms, handle multi-step authentication, and maintain session cookies exactly as a human user would.

The basic login flow is: navigate to the login page, find the username and password fields, type credentials with send_keys(), and click the submit button. After login, wait for the dashboard or landing page to load to confirm authentication succeeded. Store login state by saving cookies with driver.get_cookies() after a successful login and restoring them with driver.add_cookie() in subsequent sessions to avoid logging in every time.

Multi-factor authentication (MFA) presents a challenge for automated scraping. Some approaches include using TOTP (time-based one-time password) libraries to generate codes from a shared secret, handling email or SMS verification by integrating with email APIs, or manually completing MFA once and saving the resulting session cookies for reuse until they expire.

OAuth login flows that redirect through third-party providers (Google, Facebook, GitHub) work in Selenium because the browser follows redirects naturally. Navigate to the login page, click "Sign in with Google," handle the Google login form, and let the OAuth redirect bring you back to the target site. The session cookies will be set automatically by the browser after the OAuth flow completes.

Cookie persistence between sessions eliminates redundant login attempts. After authenticating, save all cookies to a JSON file. Before starting a new scraping session, load the cookies into the browser: navigate to the target domain first (cookies can only be set for the current domain), then add each saved cookie with driver.add_cookie(). Refresh the page to verify the session is still valid. If the cookies have expired, fall back to the full login flow.

Always respect the terms of service of any website you scrape. Authenticated scraping carries higher ethical and legal considerations than scraping public data. Use scraped data responsibly, do not overwhelm servers with requests, and avoid scraping personal data that you do not have authorization to access.

Step 5: Avoid Bot Detection

Websites use various techniques to detect and block automated browsers. Understanding these detection methods helps you configure Selenium to appear more like a genuine human visitor.

The most common detection signal is the navigator.webdriver property, which Chrome sets to true when controlled by WebDriver. Selenium 4 does not remove this flag by default, but you can use CDP commands to override it: driver.execute_cdp_cmd("Page.addScriptToEvaluateOnNewDocument", {"source": "Object.defineProperty(navigator, 'webdriver', {get: () => undefined})"}). This runs before any page JavaScript executes, hiding the automation flag.

Browser fingerprinting checks for inconsistencies between reported browser properties and actual behavior. Automated Chrome sessions often have missing plugins, zero-length plugin arrays, and inconsistent WebGL renderer strings. Libraries like undetected-chromedriver for Python patch Chrome to present a more consistent fingerprint. The library modifies ChromeDriver to remove automation-related flags and present a browser profile that matches a real user's installation.

Request patterns reveal automation. Real users browse with variable timing, take pauses to read content, and navigate somewhat randomly. Automated scrapers tend to make requests at regular intervals and follow perfectly predictable paths. Add random delays between actions, vary your navigation patterns, and avoid hitting the same endpoint hundreds of times in rapid succession.

IP-based rate limiting blocks clients that make too many requests from the same IP address. Rotating proxy services route each request through a different IP address, making it appear as though many different users are visiting the site. Residential proxies are less likely to be blocked than datacenter proxies because their IP addresses belong to real ISPs.

CAPTCHAs are the most aggressive anti-bot measure. When encountered, you can integrate with CAPTCHA-solving services that use human workers or AI to solve challenges, but this adds cost and latency. A better strategy is to configure your scraper to avoid triggering CAPTCHAs in the first place by maintaining realistic browsing patterns and rotating identifiers.

Step 6: Export and Store Scraped Data

Once you have extracted data from web pages, you need to store it in a usable format. The right storage format depends on the data structure and how you plan to use it.

CSV is the simplest format for tabular data. Python's built-in csv module writes data row by row with minimal code. Open a file, create a csv.writer, write a header row, and append data rows as you scrape each page. CSV files open directly in Excel and Google Sheets, making them accessible to non-technical stakeholders.

JSON is better for hierarchical or nested data structures. Python's json module serializes dictionaries and lists directly. Build a list of dictionaries (one per record) as you scrape, then write the entire list to a JSON file at the end. JSON preserves data types and structure that CSV flattens, and it is the standard format for feeding data into web applications and APIs.

For large scraping projects, write to a database instead of flat files. SQLite requires no server setup and stores data in a single file, which makes it ideal for local scraping projects. Python's built-in sqlite3 module handles connections, table creation, and inserts. For production scraping pipelines, PostgreSQL or MySQL provide better concurrency, data integrity, and query performance.

Pandas integration is valuable for data processing after scraping. Build a list of dictionaries during scraping, then convert to a DataFrame with pd.DataFrame(data). From there, you can clean the data (remove duplicates, fix formatting, handle missing values), perform analysis, and export to any format pandas supports, including CSV, Excel, JSON, SQL databases, and Parquet files.

Implement incremental storage to protect against crashes during long scraping runs. Instead of holding all data in memory and writing at the end, write each batch of records to disk as you go. This ensures that if the scraper crashes midway through a 10,000 page job, you do not lose the data from the pages already scraped. Append mode for CSV files and upsert operations for databases support this pattern naturally.