BeautifulSoup Tutorial for Beginners

Step 1: Install BeautifulSoup and Requests

BeautifulSoup parses HTML but does not fetch web pages on its own. You need a separate HTTP library for that, and requests is the standard choice. Open your terminal and run:

pip install beautifulsoup4 requests lxml

This installs three packages. beautifulsoup4 is the parsing library itself. requests handles HTTP requests. lxml is a fast HTML parser that BeautifulSoup can use under the hood. While BeautifulSoup works with Python's built-in html.parser, lxml is faster and handles malformed HTML better, so it is worth installing from the start.

To verify everything installed correctly, open a Python shell and run from bs4 import BeautifulSoup. If you see no errors, you are ready to go. Note the import path: you install beautifulsoup4 but import from bs4. This naming inconsistency catches many beginners, so keep it in mind.

If you want a clean environment for your project, create a virtual environment first with python -m venv venv, activate it, then install the packages. This keeps your scraping dependencies separate from your system Python. Our installation guide covers this process in more detail, including troubleshooting common installation errors.

Step 2: Fetch a Web Page

Before you can parse HTML, you need to get some. The requests library makes this straightforward. Import it, call requests.get() with a URL, and the response object contains the HTML in its .text property.

Here is a basic fetch: response = requests.get('https://example.com'). The response.status_code tells you whether the request succeeded (200 means success). The response.text contains the raw HTML string.

Always check the status code before parsing. If the server returns a 403 (forbidden) or 404 (not found), there is no point trying to parse the response. A simple guard looks like: response.raise_for_status(), which throws an exception for any HTTP error status. You can also set a custom User-Agent header by passing headers={'User-Agent': 'MyBot/1.0'} to the get call, which helps avoid blocks from servers that reject the default requests User-Agent.

For a deeper look at combining these two libraries, see our BeautifulSoup with Requests guide, which covers sessions, cookies, and handling pagination.

Step 3: Create a BeautifulSoup Object

With the HTML in hand, pass it to the BeautifulSoup constructor along with the name of the parser you want to use:

soup = BeautifulSoup(response.text, 'lxml')

This single line builds a complete parse tree from the HTML string. The soup object now represents the entire document, and every HTML tag in the page is accessible as a nested Python object. You can explore the tree immediately. For example, soup.title returns the page's <title> tag, and soup.title.string returns just the text inside it.

If you want to see the full parsed tree in a readable format, call soup.prettify(). This outputs the HTML with consistent indentation, which helps you understand the document structure before you start searching for specific elements. This is especially useful when working with pages you have not seen before.

You can also parse HTML strings directly without fetching a page. If you have raw HTML stored in a variable or a file, just pass that string to the constructor instead of response.text. This is helpful for testing and development, because you can work with a small HTML snippet without making network requests.

Step 4: Find Elements by Tag and Attribute

The two most important methods in BeautifulSoup are find() and find_all(). The find() method returns the first matching element, while find_all() returns a list of every match.

To find all links on a page: links = soup.find_all('a'). To find a specific div by its CSS class: content = soup.find('div', class_='main-content'). To find an element by its id: header = soup.find(id='page-header'). These methods accept any HTML attribute as a keyword argument, making it easy to target exactly the element you need.

The class_ parameter has an underscore because class is a reserved keyword in Python. This is the only attribute that needs the underscore. All other attributes work directly: soup.find('input', type='email'), soup.find('a', href='/about'), and so on.

You can also filter by multiple criteria at once. For example, soup.find_all('a', class_='external', href=True) finds all anchor tags that have both the "external" class and an href attribute. The href=True pattern matches any tag that has the attribute, regardless of its value. This is useful for filtering out anchor tags that are used as JavaScript triggers without real links.

For complex attribute selectors, pass a dictionary: soup.find_all('div', attrs={'data-category': 'python'}). This approach handles custom data attributes and any attribute name that is not a valid Python keyword.

Our finding elements guide goes deeper into advanced filtering, including regular expressions, custom filter functions, and searching within specific sections of a page.

Step 5: Extract Text and Attribute Values

Once you have found an element, you need to pull out its content. BeautifulSoup gives you several ways to do this depending on what you need.

For text content, .get_text() is the primary method. It extracts all text within an element, including text in nested child elements, and concatenates it into a single string. You can pass a separator: element.get_text(' ', strip=True) joins fragments with spaces and removes leading and trailing whitespace from each piece.

For a single text node with no children, .string returns the text directly. If the element has child tags, .string returns None, which makes it less versatile than get_text() but useful when you know the structure is simple.

For attribute values, use dictionary-style access: link['href'] returns the URL from an anchor tag, img['src'] returns the image source, input['value'] returns the input's value. If the attribute might not exist, use .get() to avoid a KeyError: link.get('href', '') returns an empty string when href is missing.

To get all attributes at once, access the .attrs property, which returns a standard Python dictionary. This is useful when you need to inspect what attributes are available on a tag you are unfamiliar with.

Step 6: Use CSS Selectors

If you already know CSS, BeautifulSoup's select() and select_one() methods let you use the same selector syntax you would write in a stylesheet.

soup.select('div.content p') returns all paragraphs inside any div with class "content". soup.select_one('h1#main-title') returns the first h1 with id "main-title". soup.select('ul > li:first-child') returns the first list item in every unordered list.

CSS selectors are often more concise than equivalent find_all() calls, especially when you need to express parent-child relationships or combine multiple criteria. They support class selectors (.classname), ID selectors (#idname), attribute selectors ([href], [type="text"]), descendant combinators (space), child combinators (>), and several pseudo-classes.

One practical tip: when you see a complex page structure and you are trying to figure out the right selector, use your browser's developer tools. Right-click an element, inspect it, and look at its tag, classes, and position in the DOM. Then translate that into a CSS selector for BeautifulSoup. Chrome and Firefox both let you copy a CSS selector directly from the Elements panel, which you can then paste into your select() call.

Step 7: Build a Complete Scraping Script

Now you can combine everything into a working script. A typical scraping workflow follows this pattern: define your target URL, fetch the page, parse the HTML, locate the data elements, extract values, and store the results.

Here is the structure of a basic script. You import requests and BeautifulSoup. You define the URL. You fetch the page with requests.get() and check the status. You create a BeautifulSoup object. You call find_all() or select() to get your target elements. You loop over the results, extracting text and attributes. You store the data in a list of dictionaries, which you can then write to a CSV file or load into a pandas DataFrame.

Error handling matters in real scripts. Wrap your HTTP request in a try/except block to catch connection errors and timeouts. Check that find() did not return None before trying to access attributes or text. Set a reasonable timeout on your requests with requests.get(url, timeout=10). These small practices prevent your script from crashing on unexpected pages.

For scraping multiple pages, add a loop over your list of URLs and include a delay between requests with time.sleep(1) to avoid overwhelming the server. Respect robots.txt directives and any rate limits specified by the website. Being a responsible scraper means your scripts can run reliably for longer without getting blocked.