Scraping with Requests and BeautifulSoup

Step 1: Configure Your Request Session

For any scraper that makes more than a few requests, use a requests.Session object instead of calling requests.get() directly. A Session persists cookies, headers, and connection pools across requests, which provides both performance benefits and functional advantages.

Create a session and set default headers that mimic a real browser. At minimum, set a realistic User-Agent string that matches a current version of Chrome or Firefox. Many sites block requests with the default Python Requests user-agent string. You can also set Accept, Accept-Language, and Accept-Encoding headers for additional realism.

Sessions automatically handle cookies, which is essential for scraping sites that use session tracking, CSRF tokens, or login-protected content. When you make a request through a session, any cookies set by the server are stored and sent with subsequent requests, just like a browser would.

Connection pooling is another significant benefit. A session reuses TCP connections to the same host, eliminating the overhead of establishing a new connection for each request. For scrapers that make many requests to the same domain, this can noticeably improve throughput.

Configure a retry strategy using an HTTPAdapter from the urllib3.util.retry module. Mount the adapter to your session to automatically retry failed requests with exponential backoff. This handles transient network errors and server-side 503 responses without custom retry loops in your scraping code.

Step 2: Fetch and Validate Responses

Every request your scraper makes should include a timeout parameter. Without a timeout, a request to an unresponsive server will block indefinitely, stalling your entire script. A timeout of 10 to 30 seconds works for most sites. You can pass a tuple to specify separate connect and read timeouts: timeout=(5, 15) gives 5 seconds to establish a connection and 15 seconds to receive the response.

After receiving a response, check the status code before parsing. The response.raise_for_status() method throws an exception for any 4xx or 5xx status code, which you can catch in a try/except block. Alternatively, check response.status_code directly and handle specific codes: 403 usually means bot detection, 429 means rate limiting, and 404 means the page does not exist.

Verify the content type header to confirm you received HTML rather than a redirect to a login page, a JSON error response, or a binary file. Check that response.headers.get("Content-Type", "") contains "text/html" before passing the content to BeautifulSoup.

Pay attention to character encoding. Requests guesses the encoding from HTTP headers, but sometimes guesses wrong, especially for sites that do not declare their encoding properly. If you see garbled characters in your extracted text, try setting response.encoding = "utf-8" before accessing response.text, or use response.content (raw bytes) and let BeautifulSoup handle encoding detection through its parser.

Step 3: Parse HTML with BeautifulSoup

BeautifulSoup supports multiple parsers, and the choice affects both speed and tolerance for malformed HTML. The built-in "html.parser" requires no extra installation and handles most pages well. The "lxml" parser is significantly faster and is the better choice for large documents or high-volume scraping. The "html5lib" parser is the slowest but is the most tolerant of broken HTML, parsing pages the same way a browser would.

For most projects, "lxml" provides the best balance of speed and reliability. Install it with pip install lxml and pass it as the second argument to BeautifulSoup: soup = BeautifulSoup(html, "lxml").

If you are scraping multiple pages with the same structure, define your parsing logic in a function that takes a soup object and returns a dictionary of extracted values. This keeps your code organized and makes it easy to test parsing logic independently from the fetching step. Pass saved HTML fixtures to your parsing function in tests to verify that selectors work correctly.

Step 4: Extract Data with Advanced Selectors

Basic find() and find_all() calls work for simple cases, but real-world HTML often requires more precise targeting. BeautifulSoup's select() method supports full CSS selector syntax, which gives you significantly more power.

Descendant selectors like soup.select("div.product-card h2.title") find h2 elements with the "title" class that are nested inside divs with the "product-card" class, at any depth. Child selectors using the > combinator restrict matches to direct children: soup.select("ul.menu > li") finds only top-level list items, skipping nested sub-menu items.

Attribute selectors target elements by their HTML attributes: soup.select("a[data-category='electronics']") finds links with a specific data attribute. You can also use partial attribute matching: [attr^=value] matches attributes that start with a value, [attr$=value] matches attributes that end with a value, and [attr*=value] matches attributes that contain a value anywhere.

The :nth-child() pseudo-selector helps when you need a specific element from a group. soup.select("table tr:nth-child(2) td") selects cells from the second row of a table. Combine pseudo-selectors with class and attribute filters for precise targeting.

For cases where CSS selectors are not expressive enough, use BeautifulSoup's tree navigation methods. .parent moves up one level in the tree. .next_sibling and .previous_sibling move between elements at the same level. .find_next() and .find_previous() search forward or backward in document order. These are useful for extracting data that is in a sibling or parent element relative to a label you can easily find.

Step 5: Handle Multi-Page Scraping

Most scraping projects involve multiple pages. For paginated content, locate the "next page" link on each page and follow it in a loop. Use soup.select_one("a.next-page") or a similar selector to find the next-page link, extract its href attribute, resolve it to an absolute URL using urllib.parse.urljoin(), and fetch it. Continue until no next-page link is found.

For sites that use numbered pagination parameters (like ?page=1, ?page=2), build URLs programmatically in a loop. Start at page 1 and increment until you receive an empty result set or hit a known maximum page number.

When scraping login-protected content, use the session's cookie persistence to maintain authentication. First, submit the login form by sending a POST request with the username and password as form data. The session automatically stores the authentication cookie returned by the server, and subsequent GET requests will include it. For sites with CSRF tokens, fetch the login page first, extract the token from a hidden form field, and include it in your POST data. For a dedicated guide on pagination, see How to Scrape Paginated Sites in Python.

Always add a delay between requests using time.sleep(). A random delay between 1 and 3 seconds mimics human browsing patterns better than a fixed delay and reduces the chance of triggering rate-limiting defenses.

Step 6: Clean and Structure Results

Raw scraped data is rarely ready to use immediately. Text extracted from HTML often contains extra whitespace, newline characters, non-breaking spaces, and formatting artifacts that need to be cleaned.

Start with .strip() to remove leading and trailing whitespace from every extracted string. Use .replace("\n", " ") and regular expressions to collapse internal whitespace. Remove non-breaking space characters (which appear as \xa0 in Python) with .replace("\xa0", " ").

For numeric data like prices, remove currency symbols and thousand separators before converting to float. A price displayed as "$1,299.99" needs the dollar sign and comma removed before float() can parse it.

Handle missing data explicitly. If an element does not exist on a particular page, your selector returns None, and calling .text on None raises an AttributeError. Use a conditional pattern: element.text.strip() if element else "" returns an empty string for missing elements instead of crashing.

Build your results as a list of dictionaries with consistent keys. This structure converts directly to a pandas DataFrame, CSV rows, or JSON objects. Validate each record against an expected schema before adding it to the results list. Libraries like pydantic can automate this validation, catching data type errors and missing fields before they propagate into your dataset.