How to Parse HTML in Python

Step 1: Choose a Parsing Library

Before writing any code, pick the library that best fits your project's constraints. Here is how the main options compare:

BeautifulSoup is the most popular choice for general-purpose HTML parsing. It wraps around other parsers (html.parser, lxml, html5lib) and provides a unified, beginner-friendly API. If you are starting a new project and are not sure which library to use, start here. It handles messy HTML gracefully and has excellent documentation.

lxml is a C-based library that is significantly faster than pure Python alternatives. It supports both HTML and XML parsing, provides XPath and CSS selector support, and can parse very large documents efficiently. If performance is critical or you need XPath expressions, lxml is the best choice. It can be used alone or as a parser backend for BeautifulSoup.

html.parser is Python's built-in HTML parsing module. It requires no installation, which makes it valuable in environments where you cannot install third-party packages. Its API is event-driven rather than tree-based, which means you define callback methods that fire when the parser encounters opening tags, closing tags, and text. This approach is more verbose than BeautifulSoup but requires zero dependencies.

html5lib parses HTML exactly like a web browser, following the WHATWG HTML5 specification. It is the most accurate parser for severely broken HTML but also the slowest. Use it when you need the parsed tree to match what a browser would produce.

pandas read_html is not a general HTML parser, but if your goal is extracting data from HTML tables, it is by far the fastest route. It finds all <table> elements in a document and converts them directly into DataFrames. Under the hood, it uses BeautifulSoup or lxml.

Step 2: Parse with BeautifulSoup

BeautifulSoup is the go-to library for most HTML parsing tasks in Python. Install it with pip install beautifulsoup4 lxml, then create a parser object from any HTML string:

from bs4 import BeautifulSoup
soup = BeautifulSoup(html_string, 'lxml')

The second argument specifies the parser backend. Use 'lxml' for speed, 'html.parser' for zero dependencies, or 'html5lib' for browser-identical behavior.

Finding elements is straightforward. soup.find('h1') returns the first h1 tag. soup.find_all('a', class_='external') returns all links with the "external" class. soup.select('div.content > p') uses CSS selector syntax. Extracting data uses .get_text() for text content and bracket notation for attributes: link['href'].

BeautifulSoup excels at handling imperfect HTML. It does not throw errors when tags are unclosed, attributes are malformed, or nesting is invalid. Instead, it reconstructs a usable tree from whatever markup it receives. This tolerance is critical for real-world scraping where you have no control over the HTML quality. For a deeper look at BeautifulSoup's search capabilities, see our finding elements guide.

Step 3: Parse with lxml Directly

Using lxml without the BeautifulSoup wrapper gives you maximum parsing speed and access to XPath, a powerful query language that CSS selectors cannot fully replicate.

For HTML parsing, use the lxml.html module: from lxml import html. Parse a string with tree = html.fromstring(html_string). The resulting object supports both XPath and CSS selectors.

XPath queries. tree.xpath('//div[@class="content"]//p/text()') returns all text content within paragraphs inside a content div. XPath's syntax is more verbose than CSS selectors but more powerful. It can select elements based on text content, position, and complex conditions that CSS cannot express. tree.xpath('//a[contains(@href, "example")]/@href') extracts href values from links containing "example" in the URL.

CSS selectors with lxml. Install cssselect (pip install cssselect) to use CSS selectors with lxml: tree.cssselect('div.content p'). This returns lxml Element objects, which have a different API than BeautifulSoup tags. Access text with .text and .text_content(), and attributes with .get('href') or .attrib['href'].

For XML parsing, use lxml.etree instead: from lxml import etree. This enforces XML rules and provides namespace support, which is essential for parsing RSS feeds, SOAP responses, SVG files, and other XML documents.

The trade-off for lxml's speed is a less intuitive API compared to BeautifulSoup. Operations that take one line in BeautifulSoup sometimes take two or three with lxml. For most projects, BeautifulSoup with lxml as the backend gives you the best of both worlds: BeautifulSoup's friendly API with lxml's parsing speed.

Step 4: Parse with html.parser

Python's built-in html.parser module uses an event-driven approach. Instead of building a tree and querying it, you subclass HTMLParser and override callback methods that fire as the parser encounters different elements.

Import with from html.parser import HTMLParser. Create a subclass and override handle_starttag(self, tag, attrs) for opening tags, handle_endtag(self, tag) for closing tags, and handle_data(self, data) for text content. The attrs parameter is a list of (name, value) tuples representing the tag's attributes.

This approach is more verbose and requires more manual state management. You need to track which tag you are inside, maintain your own data structures, and handle nesting yourself. However, it requires absolutely no external dependencies, which makes it suitable for constrained environments like embedded systems, minimal Docker containers, or systems where you cannot install packages.

For simple extraction tasks, like pulling all URLs from a page or extracting text from a specific section, the event-driven approach can be quite direct. For complex queries involving hierarchy and multiple conditions, BeautifulSoup or lxml is significantly more productive.

Step 5: Parse HTML Tables with pandas

If your goal is extracting tabular data from HTML, pandas.read_html() is the fastest route. It finds every <table> element in the HTML and converts each one into a pandas DataFrame in a single function call.

import pandas as pd
tables = pd.read_html('https://example.com/data')

The function accepts URLs, file paths, or raw HTML strings. It returns a list of DataFrames, one per table found. If the page has three tables, tables[0], tables[1], and tables[2] give you each one. The function automatically handles header rows, merged cells, and nested tables.

You can filter which tables to extract using the match parameter (a regex that matches against table content), the attrs parameter (a dictionary of table attributes), or the header parameter (which row to use as column headers). For example, pd.read_html(url, match='Revenue') only returns tables containing the word "Revenue".

This approach is ideal for financial data, statistics pages, sports results, scientific data tables, and any structured data that websites present in HTML table format. For non-tabular data, you still need BeautifulSoup or lxml.

Step 6: Handle Edge Cases

Broken HTML. Real-world HTML is frequently malformed. Tags are unclosed, attributes lack quotes, and nesting violates the spec. BeautifulSoup and lxml handle this gracefully by repairing the markup during parsing. html5lib does it most accurately by following the exact rules browsers use. If you encounter pages where different parsers produce different results, html5lib's output is the most trustworthy because it matches browser behavior.

Character encoding. Encoding issues manifest as garbled characters or UnicodeDecodeError exceptions. When fetching pages with the requests library, check response.encoding and verify it matches the page's actual encoding. If there is a mismatch, override it: response.encoding = 'utf-8'. When parsing HTML strings from other sources, pass bytes to BeautifulSoup and let it detect the encoding from meta tags: BeautifulSoup(raw_bytes, 'lxml').

JavaScript-rendered content. None of these libraries execute JavaScript. If the content you need is loaded dynamically by JavaScript after the initial page load, the HTML you parse will not contain it. You need a browser automation tool like Playwright or Selenium to render the page first, then pass the fully rendered HTML to your parsing library of choice.

Large documents. For HTML documents in the tens of megabytes, loading the entire document into a tree can consume excessive memory. lxml's iterparse function processes XML documents incrementally, reading only a portion at a time. For HTML, consider splitting the document or using regex to extract the specific section you need before parsing it with BeautifulSoup, reducing the amount of markup that needs to be loaded into memory.