How to Install BeautifulSoup

Step 1: Check Your Python Version

BeautifulSoup 4 requires Python 3.7 or newer. Open your terminal (Command Prompt on Windows, Terminal on macOS or Linux) and run:

python --version

If this shows Python 2.x or a "command not found" error, try python3 --version instead. On many Linux distributions and macOS, python points to Python 2 while python3 points to Python 3. You need to see something like Python 3.9.7 or higher.

If Python is not installed at all, download it from python.org. On macOS, you can also install it through Homebrew with brew install python3. On Ubuntu and Debian, use sudo apt install python3 python3-pip. On Windows, the Microsoft Store offers a one-click Python installation that automatically adds Python to your PATH.

Make sure pip is available too. Run pip --version or pip3 --version. Pip comes bundled with Python 3.4 and later, so if you have a recent Python, pip should already be there. If it is missing, run python -m ensurepip --upgrade to install it.

Step 2: Create a Virtual Environment

A virtual environment gives your project its own isolated set of Python packages. This prevents version conflicts between projects and keeps your system Python clean. While not strictly required, using a virtual environment is a strong best practice for any Python project.

Navigate to your project directory and run:

python -m venv venv

This creates a venv folder containing a self-contained Python installation. Activate it with the command for your operating system. On macOS and Linux: source venv/bin/activate. On Windows Command Prompt: venv\Scripts\activate. On Windows PowerShell: .\venv\Scripts\Activate.ps1.

When the environment is active, your terminal prompt changes to show (venv) at the beginning. All pip commands now install packages into this environment rather than your global Python installation. To leave the virtual environment later, simply run deactivate.

If you are using Anaconda or Miniconda, create a conda environment instead: conda create -n scraping python=3.11, then activate it with conda activate scraping. Conda environments work the same way conceptually but use conda's own package management system alongside pip.

Step 3: Install BeautifulSoup with pip

With your virtual environment active, install BeautifulSoup:

pip install beautifulsoup4

This downloads the latest version from PyPI and installs it along with its dependency, soupsieve, which provides CSS selector support. The installation is quick because BeautifulSoup is a pure Python package with no compiled extensions.

If you need a specific version, pin it: pip install beautifulsoup4==4.14.3. For production projects, it is good practice to pin your versions in a requirements.txt file so that every team member and deployment environment uses the same packages. Run pip freeze > requirements.txt to generate this file after installing everything you need.

On systems where both Python 2 and Python 3 are installed, you may need to use pip3 instead of pip to ensure packages install into the Python 3 environment. Inside a virtual environment, pip always points to the correct version.

Step 4: Install a Parser

BeautifulSoup does not parse HTML by itself. It delegates parsing to an underlying library, and you choose which one to use. There are three options, each with different trade-offs.

html.parser is Python's built-in parser. It requires no installation because it ships with Python itself. It is adequate for most scraping tasks, moderately fast, and handles common forms of broken HTML. If you want the simplest setup with no extra dependencies, you can skip this step and use html.parser.

lxml is a C-based parser that is significantly faster. Install it with pip install lxml. On most systems this installs a pre-compiled binary, so no C compiler is needed. If the binary is not available for your platform, pip will try to compile from source, which requires libxml2-dev and libxslt-dev on Linux. For production scraping where you process many pages, lxml is the recommended choice.

html5lib parses HTML exactly like a web browser, following the WHATWG HTML5 specification. Install it with pip install html5lib. It is the slowest parser but produces the most accurate tree for severely broken HTML. Use it when you need absolute fidelity to how browsers interpret markup.

You can install all three and switch between them by passing different parser names to the BeautifulSoup constructor. For most developers, installing lxml alongside the built-in html.parser gives you the best combination of speed and compatibility.

Step 5: Install the Requests Library

BeautifulSoup parses HTML, but it does not fetch web pages. You need a separate HTTP library, and requests is the standard choice in the Python ecosystem:

pip install requests

With requests installed, you can fetch any web page with requests.get(url) and pass the response HTML directly to BeautifulSoup. The two libraries complement each other perfectly and are almost always used together.

You can install everything at once with a single command: pip install beautifulsoup4 requests lxml. This gives you the complete stack for most web scraping projects. For more advanced HTTP handling, consider httpx as an alternative to requests, as it supports HTTP/2 and async requests.

Step 6: Verify the Installation

Open a Python shell by running python in your terminal and test the import:

from bs4 import BeautifulSoup

If no error appears, the library is installed correctly. Test parsing with a quick example:

soup = BeautifulSoup('<p>Hello</p>', 'html.parser')

print(soup.p.string)

This should print Hello. If you installed lxml, try changing the parser argument to 'lxml' to confirm that parser works too.

To verify requests is working, try fetching a test page: import requests followed by r = requests.get('https://httpbin.org/get'). If r.status_code returns 200, your full scraping stack is ready.