How to Scrape Websites with LLMs

Step 1: Set Up Your Environment

Start with Python 3.10 or later. Install the core dependencies: playwright for browser automation, openai (or anthropic) for LLM access, and beautifulsoup4 with markdownify for HTML-to-markdown conversion. Run playwright install chromium to download the browser binary.

Create a project structure with separate modules for fetching, content preparation, extraction, and validation. This separation keeps each component testable and replaceable. Store your LLM API key in an environment variable, never in source code.

Choose your LLM provider based on your accuracy and cost requirements. For development and testing, GPT-4o-mini or Claude Haiku provide fast, cheap iterations. Switch to GPT-4o or Claude Sonnet for production extractions that need higher accuracy on complex pages. If data sensitivity prevents using cloud APIs, set up a local model via Ollama with Llama 3 or Mistral.

Step 2: Fetch and Render the Target Page

Launch Playwright in headless mode and navigate to your target URL. Wait for the page to fully load, including dynamically injected content. The wait_until="networkidle" option pauses execution until all network requests settle, ensuring AJAX-loaded content is present in the DOM.

For pages with lazy loading or infinite scroll, you may need to scroll the page programmatically before capturing content. Use page.evaluate("window.scrollTo(0, document.body.scrollHeight)") and wait for new content to load. Repeat until no new content appears.

Capture the fully rendered HTML with page.content(). This gives you the complete DOM including all JavaScript-rendered elements. For pages behind login walls, add authentication steps before navigating to the target page. Playwright maintains cookies and session state between navigation actions within the same browser context.

Add proxy configuration if the target site blocks datacenter IPs. Playwright supports HTTP and SOCKS5 proxies at the browser context level. Rotate proxies between requests for large-scale operations to avoid IP-based rate limiting.

Step 3: Convert HTML to LLM-Ready Content

Raw HTML wastes tokens on markup that carries no useful information. Convert the rendered HTML to clean markdown to reduce token consumption by 80-90% while preserving content structure.

Use BeautifulSoup to extract the main content area first. Remove <nav>, <footer>, <header>, <aside>, and <script> elements. Strip elements with common boilerplate class names like "sidebar", "cookie-banner", "newsletter-signup", and "advertisement". Then use the markdownify library to convert the remaining HTML to markdown, preserving headings, lists, tables, and links.

Clean the resulting markdown by collapsing excessive whitespace, removing empty headings, and trimming trailing content that typically contains site-wide footer text. The goal is a clean document that reads like the page content a human visitor would see, without any structural or navigational noise.

Measure the token count of your prepared content. If it exceeds 80% of your model's context window (accounting for system prompt and output space), implement chunking. Split at natural boundaries like H2 headings rather than arbitrary token counts. Each chunk should be self-contained enough for the LLM to understand the context.

Step 4: Define Your Extraction Schema

The extraction schema tells the LLM exactly what structure to return. Define it as a JSON schema with field names, types, descriptions, and required/optional flags. A well-designed schema dramatically improves extraction accuracy because it gives the model clear expectations.

Keep schemas focused. Extract only the fields you actually need rather than trying to capture everything on the page. Each additional field increases the chance of errors and hallucination. Start minimal and add fields only when you confirm the model handles the existing ones reliably.

Use descriptive field names that make the expected content obvious. "product_price_usd" is clearer than "price" and reduces ambiguity when the page shows multiple monetary values (original price, sale price, shipping cost). Add field descriptions for ambiguous cases: "The current selling price, not the original retail price."

For fields that may legitimately be absent from some pages, mark them as optional and specify a null default. This prevents the model from hallucinating values to fill required fields that do not exist in the source content.

Step 5: Write and Test Extraction Prompts

The extraction prompt has three parts: a system message setting behavioral rules, the prepared page content, and the extraction instruction with your schema.

The system message should establish that the model is a precise data extraction agent. Key rules to include: only extract information explicitly present in the provided content, return null for fields not found rather than guessing, follow the exact output schema provided, and never add commentary or explanation outside the JSON structure.

Include the page content as a user message, clearly delineated with markers like "BEGIN PAGE CONTENT" and "END PAGE CONTENT." Follow it with the extraction instruction: "Extract the following fields from the page content above and return them as a JSON object matching this schema: [your schema]."

Test your prompt against 10-20 pages from your target site. Review every extraction manually to catch systematic errors. Common issues include the model confusing similar fields (extracting review count as rating), picking up sidebar content instead of main content, or formatting numbers inconsistently. Adjust your prompt wording and schema descriptions to address observed issues.

Step 6: Send Content to the LLM and Parse Results

Call the LLM API using the chat completions endpoint with your system message, page content, and extraction prompt. Set temperature=0 for maximum determinism in extraction tasks. Use response_format={"type": "json_object"} (OpenAI) or equivalent structured output mode to ensure the response is valid JSON.

Parse the JSON response and map it to your application's data model. Handle the case where the LLM returns a JSON object wrapped in markdown code fences, which happens occasionally despite structured output mode. Strip any leading/trailing text before parsing.

Log the full request and response for debugging. When extraction quality degrades on certain pages, these logs let you identify whether the issue is in content preparation (important data was stripped), prompt design (instructions are ambiguous), or model capability (the content is genuinely hard to interpret).

Step 7: Handle Errors, Retries, and Edge Cases

LLM APIs can return errors (rate limits, timeouts, server errors), malformed responses, or responses that fail schema validation. Implement exponential backoff retry logic for API errors, with a maximum of 3 retries per request.

For malformed JSON responses, try parsing with a lenient JSON parser that handles common issues like trailing commas, unquoted keys, or incomplete objects. If the response is completely unparsable, retry with a modified prompt that emphasizes the JSON format requirement more strongly.

Build validation functions for each field type. Prices should be positive numbers within a reasonable range. Dates should parse correctly. URLs should be valid. Ratings should fall within the expected scale. Flag values that pass type validation but seem implausible for human review.

For pages that exceed token limits, implement chunking with overlap. Each chunk should include 100-200 tokens of context from the previous chunk's end to prevent data loss at boundaries. After extracting from all chunks, merge results by deduplicating records based on unique identifiers (product names, URLs, or other natural keys).

Step 8: Scale Your Pipeline

Production scraping requires concurrency management, monitoring, and error tracking. Use Python's asyncio with Playwright's async API to run multiple browser pages simultaneously. Control concurrency with a semaphore to stay within your LLM provider's rate limits and your proxy pool's capacity.

Add a queue system (Redis, RabbitMQ, or even a simple database table) to manage URLs waiting for processing. This decouples URL discovery from extraction, lets you retry failed pages without restarting the entire job, and provides visibility into pipeline progress.

Monitor extraction quality continuously. Sample completed extractions for manual review, track schema validation failure rates, and alert on sudden accuracy drops that might indicate target site changes. Even though AI scrapers adapt to layout changes, major site redesigns can still affect extraction quality.

Budget for LLM costs carefully. Track cost per extraction and set spending limits. Consider using cheaper models for simple pages and reserving expensive models for pages that fail extraction with cheaper alternatives. This tiered approach can reduce total costs by 60-70% compared to using frontier models for everything.