Using Proxies with Scrapy

Understand Why Proxies Are Needed

Websites use several mechanisms to detect and block scrapers. The most common is IP-based rate limiting, where a server tracks how many requests each IP address makes within a time window. Exceeding the threshold results in HTTP 429 (Too Many Requests) responses, 403 Forbidden blocks, or CAPTCHAs.

Beyond simple rate limiting, sophisticated anti-bot systems fingerprint request patterns including request timing, header consistency, TLS fingerprints, and geographic origin. A single IP making hundreds of requests with identical headers at machine speed is an obvious automation signal. Proxies address the IP component by distributing requests across a pool of addresses, making your crawl appear as traffic from many different users.

Proxies alone do not make scraping undetectable. They are one layer in a broader strategy that includes realistic user-agent rotation, appropriate request delays, proper header sets, and respectful crawling behavior. However, without proxy rotation, even the most carefully crafted requests will eventually be blocked if the volume exceeds what one IP can sustain.

Choose a Proxy Type

Three main categories of proxies serve different scraping needs, each with distinct tradeoffs in cost, speed, and detection resistance.

Datacenter proxies are IP addresses hosted in cloud data centers. They offer the fastest speeds and lowest costs, typically a few dollars per hundred IPs monthly. The downside is that websites can easily identify datacenter IP ranges (AWS, Google Cloud, DigitalOcean) and block them preemptively. Datacenter proxies work well for sites with minimal anti-bot protection.

Residential proxies are IP addresses assigned by Internet Service Providers to real households. They appear as normal consumer traffic, making them much harder for websites to distinguish from legitimate users. Residential proxies cost more, usually billed per gigabyte of traffic at rates between $5 and $15 per GB. They are essential for scraping sites with aggressive anti-bot systems like major e-commerce platforms and social media sites.

Rotating proxy services manage large pools of proxies and automatically assign a different IP to each request or session. Services like Bright Data, Oxylabs, and SmartProxy handle the rotation logic, health monitoring, and geographic targeting on their end. You connect to a single gateway URL and each request exits through a different IP. This simplifies your Scrapy configuration at the cost of higher per-request pricing.

ISP proxies combine datacenter hosting speeds with residential IP addresses. They are static residential IPs hosted on datacenter infrastructure, offering a middle ground between speed and legitimacy. ISP proxies are useful for sites that block datacenter ranges but do not require the full anonymity of rotating residential IPs.

Configure a Single Proxy

The simplest proxy setup routes all Scrapy requests through one proxy server. This is useful for testing proxy connectivity, working with proxy services that handle rotation internally, or scraping sites where a single different IP is sufficient.

Set the proxy in your spider's request meta:

import scrapy

class ProxySpider(scrapy.Spider):
    name = "proxy_test"
    start_urls = ["https://httpbin.org/ip"]

    def start_requests(self):
        for url in self.start_urls:
            yield scrapy.Request(
                url,
                meta={"proxy": "http://user:pass@proxy.example.com:8080"},
            )

    def parse(self, response):
        self.logger.info(f"Response from: {response.text}")

Alternatively, set it globally in settings.py through environment-style configuration:

# settings.py
HTTP_PROXY = "http://user:pass@proxy.example.com:8080"
HTTPS_PROXY = "http://user:pass@proxy.example.com:8080"

Scrapy's built-in HttpProxyMiddleware reads the proxy from request meta or environment variables. For rotating proxy services that provide a gateway URL with automatic rotation, this single-proxy configuration is all you need since the service handles IP cycling on their end.

Build a Rotation Middleware

When working with a pool of proxy addresses rather than a rotating service, you need custom middleware that assigns a different proxy to each request. This distributes your traffic across the pool and reduces the request count per IP.

import random

class RotatingProxyMiddleware:
    def __init__(self, proxy_list):
        self.proxies = proxy_list
        self.current_proxy_index = 0

    @classmethod
    def from_crawler(cls, crawler):
        proxy_list = crawler.settings.getlist("PROXY_LIST")
        return cls(proxy_list)

    def process_request(self, request, spider):
        if not request.meta.get("proxy"):
            request.meta["proxy"] = random.choice(self.proxies)

    def process_response(self, request, response, spider):
        if response.status in (403, 429, 503):
            # Current proxy may be blocked, retry with different one
            new_proxy = random.choice(self.proxies)
            request.meta["proxy"] = new_proxy
            return request
        return response

    def process_exception(self, request, exception, spider):
        # Connection failed, try different proxy
        request.meta["proxy"] = random.choice(self.proxies)
        return request

Enable the middleware and configure your proxy list in settings.py:

DOWNLOADER_MIDDLEWARES = {
    "myproject.middlewares.RotatingProxyMiddleware": 350,
}

PROXY_LIST = [
    "http://user:pass@proxy1.example.com:8080",
    "http://user:pass@proxy2.example.com:8080",
    "http://user:pass@proxy3.example.com:8080",
]

The middleware priority (350) should be lower than the default HttpProxyMiddleware (750) so your custom middleware sets the proxy before Scrapy's built-in middleware processes it. The process_response method detects blocked responses and retries with a different proxy, while process_exception handles connection-level failures.

Handle Proxy Failures

Proxies fail for many reasons: the proxy server goes offline, the target site blocks that specific IP, bandwidth limits are exceeded, or authentication credentials expire. Robust proxy handling detects these failures and adapts automatically.

Common failure indicators to check in your middleware include HTTP status codes 403 (Forbidden), 407 (Proxy Authentication Required), 429 (Too Many Requests), and 503 (Service Unavailable). Also watch for CAPTCHA pages returned with 200 status codes, which require content inspection rather than status code checks.

def process_response(self, request, response, spider):
    # Check for block indicators
    if response.status in (403, 429, 503):
        return self._retry_with_new_proxy(request, spider)

    # Check for CAPTCHA or soft blocks
    if b"captcha" in response.body.lower():
        spider.logger.warning(f"CAPTCHA detected via {request.meta.get('proxy')}")
        return self._retry_with_new_proxy(request, spider)

    return response

def _retry_with_new_proxy(self, request, spider):
    retries = request.meta.get("proxy_retry_count", 0)
    if retries >= 3:
        spider.logger.error(f"Max proxy retries reached for {request.url}")
        return None
    request.meta["proxy"] = random.choice(self.proxies)
    request.meta["proxy_retry_count"] = retries + 1
    request.dont_filter = True
    return request

The dont_filter = True setting prevents Scrapy's duplicate filter from ignoring the retry, since the URL has already been seen. The retry counter prevents infinite retry loops when all proxies are blocked for a particular URL.

Optimize Proxy Performance

Proxy usage directly impacts scraping cost and speed. Several configuration choices help you get the most value from your proxy pool.

Match concurrency to pool size. If you have 50 proxies, setting CONCURRENT_REQUESTS = 50 lets each proxy handle roughly one concurrent request. Setting concurrency much higher than your pool size means multiple requests per proxy, increasing per-IP block risk.

Use sticky sessions for multi-page workflows. When your spider needs to maintain session state across multiple pages (login, add to cart, checkout), assign the same proxy to all requests in a session using request meta. Random rotation would break session cookies.

Implement geographic targeting. Some websites serve different content or block traffic based on geographic origin. If your target site is US-focused, use US-based proxies for more consistent results. Most proxy providers offer geographic filtering.

Monitor proxy health. Track success rates per proxy over time. Remove proxies that consistently fail from your active pool. Log which proxies get blocked most frequently to identify patterns, like specific datacenter subnets being blacklisted.

Balance cost and success rate. Start with cheaper datacenter proxies. If block rates exceed 10-15%, upgrade to residential proxies for the affected domains. Many projects use datacenter proxies for easy targets and residential proxies only for heavily protected sites, optimizing cost across the crawl.