How to Extract Tables from Web Pages

Step 1: Identify the Table Structure

Open the target page in a browser and use the developer tools to inspect the table element. Right-click on a table cell and select Inspect to see the underlying HTML. You are looking for several things.

Standard HTML tables use the <table> element with <thead> for headers, <tbody> for data rows, <tr> for each row, <th> for header cells, and <td> for data cells. This is the easiest structure to extract because every cell's position in the grid is explicitly defined by the markup. pandas handles these natively.

CSS-styled div tables use <div> elements styled with CSS grid or flexbox to create a visual table layout without using actual table elements. Each "cell" is a div, and the table structure exists only in the CSS rules. These require manual parsing with BeautifulSoup because pandas and other table-specific tools cannot detect them.

JavaScript data tables are rendered by libraries like DataTables, AG Grid, Handsontable, or custom React/Vue components. The HTML source may contain only an empty container element, with the table data loaded via API calls and rendered by JavaScript. Check the Network tab in developer tools to see if the raw data is fetched from an API endpoint. If it is, you can often extract the data directly from that API response rather than parsing the rendered table.

Complex table features like colspan and rowspan attributes (merged cells), nested tables, multi-level headers, and expandable row details require careful handling. Note these features during inspection so you can account for them in your extraction logic.

Step 2: Use pandas read_html for Simple Tables

The fastest way to extract standard HTML tables is with pandas.read_html(). This function takes a URL, file path, or HTML string and returns a list of DataFrames, one for each table found on the page. Install it with pip install pandas lxml html5lib (lxml and html5lib are the parsing backends that pandas uses internally).

Call dfs = pd.read_html('https://example.com/data') to extract all tables from the page. The function automatically detects table headers, handles basic colspan and rowspan merging, and converts numeric strings to appropriate data types. If the page contains multiple tables, the result is a list and you select the one you need by index.

You can refine which table is extracted by passing the match parameter with a string or regex that matches text within the target table. For example, pd.read_html(url, match='Revenue') returns only tables containing the word "Revenue." The attrs parameter lets you target tables by their HTML attributes, like attrs={'id': 'financial-data'}.

The header parameter controls which row becomes the column names. By default, pandas uses the first row with <th> elements. For tables with multi-level headers, pass a list of row indices: header=[0, 1] creates a MultiIndex column structure. The skiprows parameter lets you skip introductory rows that are not part of the data.

One limitation of read_html() is that it only works with static HTML. If the table is rendered by JavaScript, the function receives the raw HTML before JavaScript execution and finds no table data. For these cases, you need to render the page first with browser automation and then pass the rendered HTML to pandas.

Step 3: Parse Complex Tables with BeautifulSoup

When pandas cannot handle the table structure, BeautifulSoup gives you fine-grained control over the parsing process. Fetch the page HTML, create a BeautifulSoup object, find the table element, and iterate through its rows and cells.

The basic pattern is to find the table element with soup.find('table', {'id': 'target-table'}), then loop through table.find_all('tr') to get each row, and within each row, collect cell values with row.find_all(['td', 'th']). Extract the text content of each cell with cell.get_text(strip=True).

For tables with merged cells (colspan or rowspan), you need additional logic to track which cells span multiple columns or rows. Read the colspan and rowspan attributes from each cell and insert placeholder values in the appropriate positions of your output grid. This is tedious but necessary for accurate extraction from complex financial tables, timetables, and government data reports.

For div-based visual tables, identify the CSS classes or data attributes that distinguish header elements, row containers, and cell elements. Then iterate through the container divs the same way you would iterate through table rows, extracting text from each child element in order.

When tables contain links, images, or other embedded elements alongside text, decide whether you need just the visible text or additional data. Links might contain URLs you want to capture. Images might have alt text or data attributes with values. Use cell.find('a')['href'] or cell.find('img')['alt'] to extract these attributes alongside the cell text.

Step 4: Handle Dynamic and Paginated Tables

JavaScript-rendered tables require a browser automation tool to render the page before extraction. With Playwright, launch a browser, navigate to the page, and wait for the table to appear in the DOM before extracting its content.

The typical Playwright workflow for table extraction is: launch the browser with browser = playwright.chromium.launch(), create a page, navigate to the URL, wait for the table selector to appear with page.wait_for_selector('table#data'), then get the rendered HTML with page.content() and pass it to pandas or BeautifulSoup for parsing.

For paginated tables, you need to extract data from each page and combine the results. After extracting the current page, click the "Next" button (or increment the page number in the URL), wait for the new data to load, and extract again. Repeat until there are no more pages. Detect the last page by checking whether the "Next" button is disabled or absent, or by comparing the extracted data count against a total shown on the page.

Some JavaScript data tables load all their data upfront and only paginate the display client-side. In these cases, you can often change the table's "show N entries" setting to show all rows at once, then extract the complete table in a single pass. Alternatively, intercept the network requests that load the table data and read the JSON response directly, which is faster and more reliable than parsing rendered HTML.

Infinite-scroll tables require automating the scroll action. Use Playwright's page.evaluate('window.scrollTo(0, document.body.scrollHeight)') in a loop, waiting for new rows to appear after each scroll, until the row count stops increasing.

Step 5: Clean and Export the Data

Raw extracted table data usually needs cleaning before it is useful. Common issues include inconsistent column names (leading or trailing spaces, mixed capitalization), empty rows from visual separators, merged header rows that need to be collapsed, numeric values with formatting characters (dollar signs, commas, percent symbols), and mixed data types within columns.

Normalize column names by stripping whitespace, converting to lowercase, and replacing spaces with underscores. Remove rows that contain only empty cells or visual separators. Convert numeric columns by stripping formatting characters and casting to int or float types. Parse date strings into datetime objects using a consistent format.

Handle missing values explicitly. Decide whether empty cells should become None, NaN, zero, or an empty string based on the context. Some tables use dashes, "N/A", or "n/a" to indicate missing values, and these should be standardized to a single representation.

Export to the format your downstream process needs. df.to_csv('output.csv', index=False) creates a CSV file. df.to_json('output.json', orient='records') creates a JSON array of objects. For database storage, use df.to_sql('table_name', connection) with a SQLAlchemy connection to write directly to PostgreSQL, MySQL, or SQLite.