How to Extract Data from PDFs

Step 1: Determine the PDF Type

PDFs fall into three categories, and the extraction method depends entirely on which type you are working with.

Native digital PDFs are created by software like Microsoft Word, Google Docs, LaTeX, or reporting tools. The text in these files is stored as actual character data that extraction tools can read directly. You can verify this by opening the PDF and trying to select and copy text. If the text highlights cleanly and pastes correctly, you have a native PDF.

Scanned PDFs are images of paper documents that have been run through a scanner or photographed. The file contains bitmap images of pages, not text characters. If you cannot select individual words in the PDF viewer, you are dealing with a scanned document. These require optical character recognition (OCR) to convert the images into machine-readable text before any data extraction can begin.

Hybrid PDFs contain a mix of native text and scanned images. A common example is a digitally generated report that includes a scanned signature page or embedded images of handwritten notes. Hybrid documents require both text extraction (for the native portions) and OCR (for the scanned portions), which makes them the most complex to process reliably.

You can detect the PDF type programmatically by attempting to extract text from a page. If the extraction returns meaningful text, the page is native. If it returns empty or garbled output, the page likely contains a scanned image and needs OCR.

Step 2: Choose Your Extraction Tool

For native PDF text extraction, several Python libraries handle the job well.

pdfplumber is the most capable general-purpose PDF extraction library in Python. It extracts text with positional information (coordinates for each character), detects and extracts tables, and handles multi-column layouts. It works by analyzing the character positions in the PDF to reconstruct words, lines, and paragraphs. pdfplumber is the recommended starting point for most PDF extraction tasks because of its accuracy and its ability to handle both text and tables.

PyPDF2 (now maintained as pypdf) is a lighter-weight option that handles basic text extraction, page manipulation, and metadata reading. It is faster than pdfplumber for simple text extraction but less accurate with complex layouts and does not have built-in table detection. Use PyPDF2 when you need to extract plain text from well-structured documents.

Camelot specializes in table extraction from PDFs. It uses two methods: lattice mode (detecting table borders drawn as lines) and stream mode (detecting tables from text alignment patterns). Camelot outputs tables as pandas DataFrames, making it easy to analyze and export the data. If your primary goal is extracting tabular data, Camelot is the best dedicated tool for the job.

Tabula (via tabula-py) is another table extraction tool that wraps the Java-based Tabula library. It works similarly to Camelot but uses a different detection algorithm. Some documents extract better with Tabula than Camelot and vice versa, so having both available gives you options when one tool struggles with a particular layout.

For scanned PDFs, you need an OCR engine.

Tesseract is the leading open-source OCR engine, maintained by Google. The Python wrapper pytesseract makes it easy to integrate into extraction pipelines. Tesseract supports over 100 languages and achieves good accuracy on clean, well-scanned documents. Accuracy drops with poor scan quality, unusual fonts, or handwritten text. For best results, preprocess images to improve contrast, remove noise, and deskew rotated pages before running OCR.

Amazon Textract is a cloud service that goes beyond basic OCR by understanding document structure. It identifies forms (key-value pairs), tables (with rows and columns), and different content blocks on each page. Textract handles complex document layouts that Tesseract struggles with, but it requires an AWS account and charges per page processed. For production workloads processing thousands of documents, Textract's accuracy and structure awareness often justify the cost.

Google Document AI offers similar cloud-based document understanding with specialized processors for invoices, receipts, bank statements, and other common document types. Its custom processor training lets you build extraction models tailored to your specific document formats.

Step 3: Extract Text Content

With pdfplumber, extracting text from a native PDF is straightforward. Open the file, iterate over each page, and call the extract_text() method. The library reconstructs the reading order from character positions and returns the text as a string with newlines separating paragraphs.

For multi-column documents, pdfplumber's default text extraction may interleave text from adjacent columns. You can handle this by using the extract_words() method to get individual words with their coordinates, then grouping words by their horizontal position to reconstruct each column separately. Define column boundaries based on the x-coordinates of the words and process each column independently.

Filter out page headers, footers, and page numbers that repeat across pages. These are typically located in consistent positions (top or bottom of the page area) and contain the same or similar text on every page. Detect them by comparing text in the same coordinate ranges across multiple pages and excluding any text that appears in the same position on more than half the pages.

For scanned PDFs, the process starts with converting each page to an image (using a library like pdf2image), then running OCR on each image. Tesseract returns the recognized text, which you can then process the same way as native text. The additional OCR step adds processing time and may introduce recognition errors that need to be handled in the cleaning stage.

Step 4: Extract Tables

Table extraction is the most challenging part of PDF processing because tables rely on spatial alignment rather than explicit structural markup. A cell boundary in a PDF might be drawn as four separate line segments, or it might exist only implicitly through whitespace alignment.

With Camelot, call camelot.read_pdf(filepath, pages='all', flavor='lattice') for tables with visible borders or flavor='stream' for tables without borders. Camelot returns a list of Table objects, each containing a pandas DataFrame with the extracted data. Check the parsing accuracy by examining the table's parsing_report, which includes a whitespace metric indicating how well the table was detected.

For PDFs where neither Camelot nor Tabula produces clean results, consider a two-step approach: use pdfplumber to extract all text with coordinates, then write custom logic to group text elements into rows and columns based on their spatial positions. This approach requires more code but gives you full control over how the table structure is interpreted.

When working with tables from web pages, the HTML table structure makes extraction much simpler than PDF tables. If the source data is available on a website as well as in PDF form, extracting from the web version is almost always easier and more reliable.

Step 5: Clean and Structure the Output

Raw extracted text contains artifacts that need cleaning before the data is usable. Common issues include extra whitespace from column layouts, hyphenated words split across lines, OCR errors like confusing "l" with "1" or "O" with "0", and encoding problems with special characters.

Build a cleaning pipeline that processes the raw text through a series of normalization steps: collapse multiple spaces into single spaces, rejoin hyphenated words that were split across lines, correct common OCR substitution errors, and standardize date formats, currency values, and number formatting.

Map the cleaned text to structured fields based on your extraction requirements. If you are extracting invoices, identify and capture the invoice number, date, vendor name, line items, quantities, unit prices, and totals. If you are extracting research papers, capture the title, authors, abstract, section headings, and references. Define a schema for your output and validate each extracted record against it.

For high-volume document processing, consider using an AI-powered extraction platform that handles the cleaning and structuring automatically. These platforms use machine learning models trained on thousands of similar documents to achieve higher accuracy than rule-based approaches, especially for documents with varied layouts.