How to Scrape a Website

Step 1: Choose Your Scraping Approach

Your first decision is whether the target website renders content on the server (as static HTML) or on the client (using JavaScript). Open the target page in your browser, right-click, and select "View Page Source." If the data you want appears in the raw HTML source, a simple HTTP client library like Python's requests or Node.js axios paired with a parser will work. This is the fastest and most resource-efficient approach.

If the page source shows placeholder elements, loading spinners, or JavaScript variables instead of actual content, the site renders dynamically and you will need a headless browser. Playwright is the current gold standard for headless scraping because it supports Chromium, Firefox, and WebKit, offers built-in wait conditions, and provides a clean async API. Puppeteer is a strong alternative if you prefer a Chromium-only tool with a slightly simpler API surface. Selenium remains widely used, especially in environments that already depend on it for testing.

For quick, one-off extraction tasks on sites with straightforward HTML, a browser extension can get the job done without writing code at all. Extensions work well when you need data from a handful of pages but become impractical for large-scale, recurring collection.

Step 2: Inspect the Target Page Structure

Open the target page in Chrome or Firefox and launch the developer tools (F12 or Ctrl+Shift+I). Use the element inspector to click on the data you want to extract. The Elements panel will highlight the corresponding HTML tag and show its class names, ID, attributes, and position in the DOM tree.

Look for patterns that repeat across the data elements you want to collect. On a product listing page, for example, each product typically lives inside a container element (like a <div class="product-card">), and within that container, the name, price, image, and rating each occupy predictable child elements with consistent class names. Identifying these patterns is the foundation of reliable scraping.

Pay attention to data attributes, ARIA labels, and semantic HTML elements, as these can provide more stable selectors than class names, which often change during redesigns or A/B tests. An attribute like data-product-id or itemprop="price" is usually more durable than a generated class name like css-1a2b3c.

Also note whether the site uses iframes, shadow DOM elements, or web components, as these create boundaries that standard CSS selectors cannot cross without additional handling.

Step 3: Check robots.txt and Terms of Service

Before you start scraping, visit the target site's robots.txt file (usually at the root URL, like example.com/robots.txt). This file specifies which paths are allowed or disallowed for automated agents, and it may include a Crawl-delay directive indicating how many seconds to wait between requests. While robots.txt is not legally binding in most jurisdictions, respecting it is considered best practice and demonstrates good faith.

Review the site's terms of service for language about automated access, data collection, or bot usage. Many commercial sites explicitly restrict scraping in their terms. Understanding these restrictions before you start helps you make informed decisions about whether to proceed and how to do so responsibly. For a detailed analysis of the legal landscape, see our guide on whether web scraping is legal.

Step 4: Write Your Selectors and Extraction Logic

With the page structure understood, write CSS selectors or XPath expressions that isolate your target elements. Start broad and refine. For example, if product prices live in <span class="price" data-currency="USD"> elements, the selector span.price[data-currency="USD"] will match them precisely.

Test your selectors in the browser console before putting them into code. In Chrome's console, document.querySelectorAll('your-selector') returns all matching elements, letting you verify accuracy instantly. For XPath, use document.evaluate() or the $x() console shortcut.

Your extraction code should iterate over all matched elements and pull the text content, attribute values, or inner HTML from each one. Clean the extracted text by stripping whitespace, removing currency symbols or units if you need numeric values, and normalizing formats (like date strings) to a consistent representation. Build a data structure (a list of dictionaries in Python, an array of objects in JavaScript) that represents one complete record per item.

Handle edge cases from the start: some items may be missing a price (out of stock), have multiple variants, or display "Call for pricing" instead of a number. Write conditional logic that captures what is available and marks missing fields rather than crashing on unexpected structures.

Step 5: Handle Pagination and Navigation

Most datasets of interest span multiple pages. Identify how the target site implements pagination. Common patterns include numbered page links (?page=2, ?page=3), "Next" buttons that advance to the next result set, "Load More" buttons that fetch additional items via AJAX, and infinite scroll that triggers API calls as you scroll down the page.

For URL-parameter pagination, construct the full list of page URLs upfront if you know the total page count, or increment the page number until you receive an empty result set. For button-based pagination with a headless browser, locate and click the "Next" or "Load More" element, wait for the new content to appear, then extract the additional items. For infinite scroll, programmatically scroll the page and wait for new elements to load, repeating until no more content appears.

Implement a reasonable delay between page requests (typically 1 to 3 seconds) to avoid overwhelming the server and triggering rate limits. Randomizing the delay slightly makes your request pattern look more natural to anti-bot systems.

Step 6: Add Error Handling and Rate Limiting

Production scrapers encounter failures regularly. Network connections timeout, servers return 500 errors, pages display unexpected layouts, and anti-bot systems return 403 or 429 responses. Build resilience into your scraper from the beginning.

Implement exponential backoff for retries: if a request fails, wait 2 seconds before retrying, then 4 seconds, then 8, up to a maximum retry count. This pattern avoids hammering a struggling server while giving transient errors time to resolve. Log every failure with the URL, status code, and timestamp so you can review and fix systematic issues later.

Validate extracted data against expected patterns. If a price field contains letters instead of numbers, or a date field is empty when it should always have a value, log a warning rather than silently storing bad data. Data validation catches both extraction bugs and website changes early, before they contaminate your dataset.

Set a polite request rate that respects the target server's capacity. A good starting point is one request per second for small sites and two to five concurrent requests for large, well-resourced platforms. Monitor response times and back off if they increase, as rising latency often signals server strain.

Step 7: Store and Export Your Data

Choose a storage format that matches your downstream use case. CSV works well for tabular data that will be analyzed in spreadsheets or imported into business intelligence tools. JSON is better suited for nested or hierarchical data structures and integrates naturally with web applications and APIs. For production systems that need querying, filtering, and joining across datasets, write directly to a database like PostgreSQL, SQLite, or MongoDB.

Apply any final data cleaning or transformation before storage. Deduplicate records if your scraper might encounter the same item on multiple pages. Normalize text fields (trim whitespace, standardize capitalization, convert encoded characters). Convert numeric strings to actual numbers and date strings to standard date formats.

If you plan to run the scraper on a recurring schedule, implement incremental scraping that only collects new or changed records rather than re-scraping the entire dataset each time. Store a record ID or hash from previous runs and skip items that match, reducing load on both the target server and your own infrastructure.