BeautifulSoup with Requests

Step 1: Fetch a Page with Requests

The basic workflow starts with a single function call. Import requests, call requests.get() with your target URL, and the response object gives you everything you need.

response = requests.get('https://example.com')

The response object contains several useful properties. response.status_code tells you whether the request succeeded (200 for success, 404 for not found, 403 for forbidden, and so on). response.text contains the HTML as a decoded string. response.content contains the raw bytes, which matters when the page uses an unusual character encoding. response.headers contains the server's response headers as a dictionary.

Always set a timeout to prevent your script from hanging indefinitely on slow or unresponsive servers: requests.get(url, timeout=10). The timeout value is in seconds. Without a timeout, a request to a server that never responds will block your script forever.

For pages that require query parameters, pass them as a dictionary: requests.get(url, params={'q': 'python', 'page': '2'}). The requests library properly URL-encodes the parameters and appends them to the URL, which is cleaner and more reliable than building the query string by hand.

Step 2: Parse the Response with BeautifulSoup

Once you have the HTML, pass it to BeautifulSoup along with your preferred parser:

soup = BeautifulSoup(response.text, 'lxml')

The soup object is now a fully navigable parse tree. You can call soup.find(), soup.find_all(), soup.select(), or any other BeautifulSoup method to locate and extract data. Our finding elements guide covers these methods in depth.

Use response.text (the decoded string) rather than response.content (raw bytes) in most cases. The requests library automatically detects the page's character encoding from the HTTP headers and decodes the content accordingly. If you encounter encoding issues, such as garbled characters in the parsed output, you can check response.encoding and override it: response.encoding = 'utf-8' before accessing response.text.

For pages with known encoding issues, you can also pass the raw bytes to BeautifulSoup and let the parser handle encoding detection: BeautifulSoup(response.content, 'lxml'). The lxml parser is particularly good at detecting encodings from meta tags within the HTML itself.

Step 3: Set Headers and User-Agent

Many websites check the User-Agent header to identify the client making the request. The default requests User-Agent is python-requests/x.x.x, which some servers block because it identifies automated traffic. Setting a browser-like User-Agent helps avoid these blocks.

Pass headers as a dictionary: requests.get(url, headers={'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36'}). This tells the server your request is coming from a standard browser rather than a script.

Beyond User-Agent, you can set other headers that browsers normally send. Accept tells the server what content types you can handle. Accept-Language requests content in a specific language. Referer indicates which page linked to the current request. Some sites check these headers for consistency and block requests that are missing standard browser headers.

If you are making many requests, define your headers once in a dictionary and reuse them, or better yet, use a Session object (covered in the next step) to persist headers across all requests automatically.

Step 4: Use Sessions for Cookies and Authentication

A requests.Session object persists settings across multiple requests. This is essential when scraping sites that require login, use CSRF tokens, or track visitors with cookies.

Create a session with session = requests.Session(). Set default headers on the session: session.headers.update({'User-Agent': '...'}). Now every request made through session.get() or session.post() includes those headers and carries forward any cookies set by previous responses.

Login-protected pages. To scrape pages behind a login, first POST the credentials to the login endpoint: session.post('https://example.com/login', data={'username': 'user', 'password': 'pass'}). The session stores the authentication cookies from the response. Subsequent session.get() calls include those cookies automatically, giving you access to protected pages without re-authenticating.

CSRF tokens. Many sites include a hidden CSRF token in their forms that must be submitted with POST requests. The workflow is: fetch the login page with GET, parse the form with BeautifulSoup to extract the token value, then include it in your POST data. The session handles the cookies while BeautifulSoup handles the token extraction.

Connection pooling. Sessions also reuse the underlying TCP connection for requests to the same host, which improves performance when making many requests to the same site. This connection pooling happens transparently without any configuration.

Step 5: Handle Pagination

Most sites spread data across multiple pages. There are two common pagination patterns, and your scraping approach differs for each one.

URL-based pagination uses query parameters or path segments to identify page numbers. URLs look like example.com/results?page=1, example.com/results?page=2, and so on. To scrape these, loop through page numbers and construct each URL: iterate from 1 to the last page number, build the URL with the current page parameter, fetch and parse each page, and collect the results.

To find the total number of pages, parse the first page and look for pagination controls. Many sites show "Page 1 of 25" or have a last-page link. Extract that number with BeautifulSoup and use it as your loop boundary.

Next-link pagination uses "Next" buttons that link to the following page. The advantage is you do not need to know the total page count in advance. Start with the first page URL. On each page, parse the HTML and look for the next-page link. If it exists, fetch that URL and repeat. If it does not exist, you have reached the last page. This pattern handles dynamic pagination and load-more buttons well, though it requires sequential processing since each page depends on the previous one.

Whichever pattern you use, add a delay between requests. time.sleep(1) pauses for one second between page fetches, which prevents overwhelming the server and reduces the chance of getting blocked. For large scraping jobs, consider randomizing the delay slightly to make the request pattern less predictable.

Step 6: Add Error Handling and Rate Limiting

Production scraping scripts need to handle failures gracefully. Network errors, server errors, and unexpected page structures can all interrupt your scraping run.

HTTP error checking. After every request, check the status code. response.raise_for_status() throws an exception for any 4xx or 5xx response. Alternatively, check response.status_code directly and handle specific codes: retry on 429 (too many requests) and 503 (service unavailable), skip on 404 (not found), and abort on 403 (forbidden).

Network error handling. Wrap your requests in try-except blocks to catch requests.exceptions.ConnectionError (server unreachable), requests.exceptions.Timeout (request took too long), and requests.exceptions.RequestException (catch-all for any request error). Log the error and either retry or skip the URL depending on your needs.

Retries. The requests library supports automatic retries through the urllib3.util.retry.Retry class and requests.adapters.HTTPAdapter. Mount a retry adapter on your session with configurable retry counts, backoff factors, and the HTTP status codes that trigger retries. This is cleaner than writing your own retry loop and handles exponential backoff automatically.

Rate limiting. Respect the server you are scraping by adding delays between requests. time.sleep(1) is a reasonable starting point. If a server returns 429 (Too Many Requests), it may include a Retry-After header telling you how long to wait. Honor that value. For large-scale scraping, consider a token bucket or leaky bucket rate limiter for more precise control over request frequency.

Parsing safety. Always check that find() did not return None before accessing properties. Page structures change without warning, and a missing element should not crash your entire scraping run. Use conditional checks or try-except blocks around extraction code, and log pages where expected elements are missing so you can investigate later.