Python Web Scraping Tutorial

Step 1: Set Up Your Environment

Before writing any scraping code, you need a properly configured Python environment. Start by confirming you have Python 3.8 or later installed on your system. Open a terminal and run python --version or python3 --version to check. If Python is not installed, download it from the official python.org website.

Next, create a virtual environment to keep your project dependencies isolated from your system Python installation. Run python -m venv scraper-env to create the environment, then activate it with source scraper-env/bin/activate on macOS and Linux, or scraper-env\Scripts\activate on Windows.

With the virtual environment active, install the two core libraries. Run pip install requests beautifulsoup4 in your terminal. Requests handles HTTP communication, sending requests to web servers and receiving their responses. BeautifulSoup handles HTML parsing, converting raw HTML text into a searchable tree structure. These two libraries together cover the full scraping workflow for static websites.

Create a new Python file called scraper.py in your project directory. At the top, add your import statements: import requests and from bs4 import BeautifulSoup. You are now ready to start writing your scraper.

Step 2: Fetch a Web Page

The first task in any scraping script is downloading the HTML content of the target page. The Requests library makes this straightforward with its get() function. You pass in the URL of the page you want to scrape, and Requests returns a Response object containing the server's reply.

A basic fetch looks like this: you assign the result of requests.get("https://example.com") to a variable, typically called response. The Response object has several useful attributes. response.status_code tells you whether the request succeeded (200 means success). response.text contains the raw HTML of the page as a string. response.headers contains the HTTP headers returned by the server.

Always check the status code before proceeding. If the status code is not 200, something went wrong. A 403 means the server blocked your request, often because it detected an automated client. A 404 means the page does not exist. A 429 means you are sending requests too quickly and the server is rate-limiting you. A 500 means the server encountered an internal error.

For more reliable requests, set a timeout and include realistic headers. The timeout prevents your script from hanging indefinitely if a server does not respond. Adding a User-Agent header that matches a real browser makes your request less likely to be blocked. A common pattern is to create a headers dictionary with a User-Agent string from a modern Chrome or Firefox browser, then pass it to the get() call along with a timeout of 10 to 30 seconds.

Step 3: Parse the HTML

Once you have the raw HTML in response.text, you need to convert it into a structure that you can search and navigate programmatically. This is where BeautifulSoup comes in.

Create a BeautifulSoup object by passing the HTML text and a parser name to its constructor: soup = BeautifulSoup(response.text, "html.parser"). The "html.parser" argument tells BeautifulSoup to use Python's built-in HTML parser. If you need faster performance, you can install and use "lxml" instead, which parses large documents significantly faster.

The soup object now represents the entire HTML document as a tree of nested elements. You can navigate this tree using tag names, attributes, text content, and CSS selectors. BeautifulSoup handles malformed HTML gracefully, automatically fixing unclosed tags and other issues that are common on real-world websites.

To verify that parsing worked correctly, you can print a portion of the parsed tree. Calling soup.title returns the page's title tag, and soup.title.string returns just the text inside it. If these return sensible values, your fetch and parse steps are working correctly.

Step 4: Locate Target Elements

With the parsed HTML in hand, you need to identify which elements contain the data you want. BeautifulSoup provides several methods for finding elements.

The find() method returns the first element matching your criteria. For example, soup.find("h1") returns the first h1 tag on the page. You can also search by CSS class using the class_ parameter: soup.find("div", class_="product-info") returns the first div with that class name.

The find_all() method returns a list of all matching elements. soup.find_all("a") returns every link on the page. soup.find_all("p", class_="description") returns every paragraph with the "description" class. You can limit results with the limit parameter: soup.find_all("li", limit=10) returns only the first 10 list items.

The select() method uses CSS selector syntax, which is often more intuitive for developers familiar with CSS or jQuery. soup.select("div.product h2") finds all h2 elements inside divs with the "product" class. soup.select("#main-content .price") finds elements with the "price" class inside the element with the "main-content" ID. CSS selectors are especially useful for navigating complex nested structures.

To figure out which selectors to use, open the target page in your web browser, right-click on the data you want, and choose "Inspect" or "Inspect Element" from the context menu. The browser's developer tools will show you the HTML structure, including tag names, classes, IDs, and nesting depth. Use this information to write selectors that precisely target the data you need.

Step 5: Extract Data from Elements

Once you have located the right elements, you need to pull the actual data values out of them. BeautifulSoup elements expose their content through several properties and methods.

The .text or .get_text() property returns the visible text content of an element, stripping all HTML tags. If you have a tag like <span class="price">$29.99</span>, calling .text on it returns the string "$29.99". The get_text() method accepts optional parameters for adding separators between text from nested elements and for stripping whitespace.

To extract attribute values, use dictionary-style access or the .get() method. For a link element, element["href"] returns the URL. For an image, element["src"] returns the image source. Using .get() is safer because it returns None instead of raising an error if the attribute does not exist: element.get("data-id", "") returns an empty string as a default if the data-id attribute is missing.

When scraping multiple items (such as a list of products or articles), iterate over the results of find_all() or select() and extract data from each element. A common pattern is to build a list of dictionaries, where each dictionary represents one item with its extracted fields. This structure converts naturally to a pandas DataFrame or CSV rows for later processing.

Clean the extracted text as you go. Use .strip() to remove leading and trailing whitespace, and Python string methods like .replace() to handle special characters, currency symbols, or formatting artifacts. Consistent cleaning at extraction time prevents data quality issues downstream.

Step 6: Handle Errors and Edge Cases

Production scrapers encounter errors frequently, and your code needs to handle them gracefully rather than crashing. The most common error sources are network failures, unexpected HTML structures, and server-side blocking.

Wrap your request calls in try/except blocks to catch requests.exceptions.RequestException, which is the base class for all Requests errors including connection failures, DNS resolution errors, and timeouts. Log the error details so you can diagnose problems later, and decide whether to retry the request or skip that page and continue with others.

When parsing, always account for the possibility that an element you expect might not exist. If a product listing is missing a price element, calling .text on a None result will crash your script. Check for None before accessing properties: use Python's conditional expression (element.text if element else "N/A") or try/except blocks around extraction code.

Implement retry logic for transient errors. If a request fails with a timeout or 503 error, waiting a few seconds and retrying often succeeds. A simple retry loop with exponential backoff (waiting 2 seconds, then 4, then 8) handles most transient issues. The urllib3 library, which Requests uses internally, also provides built-in retry functionality that you can configure through a Session adapter.

Add a delay between requests using time.sleep() to avoid overwhelming the target server. Even a one-second delay dramatically reduces the chance of being rate-limited or blocked. For respectful scraping, check the target site's robots.txt for a Crawl-delay directive and honor it.

Step 7: Save Results to a File

The final step is persisting your extracted data so you can use it later. For most scraping projects, CSV and JSON are the two most practical output formats.

To save as CSV, use Python's built-in csv module. Open a file in write mode, create a csv.DictWriter with your field names, call writeheader() to write the column names, and then call writerow() or writerows() to write the data. Specify encoding="utf-8" and newline="" when opening the file to ensure consistent formatting across operating systems.

To save as JSON, use Python's built-in json module. Call json.dump() with your data and a file object. Add indent=2 for readable formatting and ensure_ascii=False if your data contains non-ASCII characters. JSON is especially useful when your data has nested structures that do not fit cleanly into CSV rows.

For larger projects, consider writing results to a database. SQLite requires no server setup and is included in Python's standard library. Create a connection, define a table schema, and insert rows as you scrape. This approach handles millions of rows efficiently and lets you query your data with SQL. For a detailed guide on CSV export specifically, see How to Export Scraped Data to CSV.