How to Use Proxies in Python Scraping
Python dominates the web scraping ecosystem because of its rich library support and readable syntax. Every major scraping library supports proxy configuration, but the implementation details differ significantly between simple HTTP clients, full scraping frameworks, and browser automation tools. Understanding the right approach for each tool prevents common integration mistakes and lets you build robust proxy handling from the start.
Configure Proxies With the Requests Library
The Requests library is the most common starting point for Python scraping. It accepts proxy configuration through a simple dictionary that maps protocol schemes to proxy URLs. For a basic HTTP proxy, you pass a dictionary with "http" and "https" keys pointing to your proxy address including the port number.
The proxy URL format follows the pattern protocol://username:password@host:port for authenticated proxies, or protocol://host:port for IP-whitelisted proxies that do not require credentials. When using HTTPS proxies, the scheme in the dictionary key must be "https" and the proxy URL itself should use "http://" as the connection protocol to the proxy server (the proxy then handles the HTTPS tunnel to the target).
For persistent proxy usage across multiple requests, create a Session object and set its proxies attribute once. All subsequent requests through that session automatically use the configured proxy without needing to pass the dictionary on each call. Sessions also maintain cookies and connection pooling, which improves performance when making many sequential requests through the same proxy.
To rotate proxies with Requests, maintain a list of proxy addresses and select a different one for each request. You can implement this as a simple function that picks a random proxy from your pool, or use a more sophisticated manager that tracks which proxies are healthy and which should be in cooldown. The key is that each call to requests.get() or session.get() can accept a fresh proxies dictionary, allowing per-request proxy selection.
Set Up Proxy Middleware in Scrapy
Scrapy handles proxies through its middleware system, which processes requests between your spider logic and the network layer. The simplest approach sets the proxy on each request by assigning the request.meta["proxy"] attribute in your spider. However, for rotating proxies at scale, a custom downloader middleware provides cleaner architecture.
A proxy rotation middleware intercepts every request before it leaves Scrapy's engine and assigns a proxy address from your pool. The middleware class implements the process_request method, which receives each Request object and can modify it before transmission. Inside this method, you select the next proxy from your rotation logic and set request.meta["proxy"] to the proxy URL.
For authenticated proxies in Scrapy, you need to set both the proxy URL and the Proxy-Authorization header. The header value is "Basic" followed by the base64-encoded username:password string. Alternatively, include credentials directly in the proxy URL (http://user:pass@host:port) and Scrapy handles the authorization header automatically.
Scrapy's built-in retry middleware works well with proxy rotation. When a request fails through one proxy, the retry middleware requeues it, and your proxy middleware assigns a different proxy on the retry attempt. This combination provides automatic failover without custom retry logic. Configure RETRY_TIMES in your settings to control how many proxy attempts each URL gets before being marked as failed.
Configure Browser Proxies With Playwright
Playwright configures proxies at the browser launch level, meaning all traffic from that browser instance routes through the specified proxy. The proxy settings go in the launch options dictionary with server, username, and password fields. Unlike Requests where you can change proxies per-request, Playwright locks the proxy for the entire browser context.
To rotate proxies with Playwright, you launch a new browser context for each proxy IP rather than trying to switch mid-session. Create the browser with no proxy, then create individual contexts with different proxy settings. Each context operates as an isolated browser with its own cookies, storage, and proxy configuration. When you need a new IP, close the current context and create a fresh one with the next proxy in your rotation.
For Playwright's async API (which is recommended for scraping), the pattern involves creating a pool of browser contexts, each configured with a different proxy. Your scraping logic acquires a context from the pool, navigates to the target page, extracts data, then returns the context for reuse or closes it and creates a replacement with a fresh proxy. This pooling approach amortizes the browser launch cost across many requests.
SOCKS5 proxies work with Playwright by specifying "socks5://host:port" in the server field. This is useful when your proxy provider offers SOCKS5 endpoints, which some providers prefer for browser traffic because SOCKS5 handles all protocol types without needing separate HTTP and HTTPS proxy configurations.
Implement Proxy Rotation Logic
A production proxy rotation system needs more intelligence than random selection from a list. Build a ProxyManager class that maintains state about each proxy's health, recent usage, and performance metrics. The class should track success count, failure count, last used timestamp, cooldown expiration, and average response time for each proxy in the pool.
The get_proxy() method should implement weighted selection that prefers proxies with high success rates, long idle times since last use, and no active cooldown periods. When a proxy fails, call a mark_failed() method that increments its failure counter and potentially puts it in cooldown. When a proxy succeeds, call mark_success() to update its health score positively.
Implement automatic cooldown with exponential backoff. The first failure triggers a short cooldown (30 seconds), the second consecutive failure doubles it (60 seconds), and so on up to a maximum cooldown period (1 hour or permanent removal). This prevents repeatedly routing traffic through a proxy that the target has clearly blocked while still allowing temporarily rate-limited proxies to recover.
For thread-safe operation in concurrent scrapers, protect your proxy pool state with locks or use thread-safe data structures. Multiple scraping threads calling get_proxy() simultaneously should not receive the same proxy (unless your pool is large enough that collisions are acceptable) and should not corrupt the internal state tracking.
Handle Proxy Authentication and Errors
Proxy authentication failures manifest differently than target website blocks. A 407 Proxy Authentication Required response means your credentials are wrong or expired, not that the target blocked you. Handle this separately from target-level errors by checking the HTTP status code and response headers to determine whether the proxy itself or the target website generated the error.
Common proxy errors include connection timeouts (the proxy server is unreachable), tunnel failures (the proxy cannot connect to the target), and bandwidth exceeded errors (you have used your monthly allocation). Each requires different handling: timeouts should trigger proxy rotation, tunnel failures might indicate the target is blocking the proxy's IP at the network level, and bandwidth errors require stopping all requests until the allocation resets.
Implement a response validation layer that checks whether you received the expected content type and structure. Some anti-bot systems return 200 OK with a CAPTCHA page or redirect page rather than a proper error code. Your validation should confirm that the response contains actual target content (checking for expected HTML structure, known page elements, or minimum content length) before considering the request successful.
Log proxy performance metrics continuously. Track success rate per proxy, per target domain, and per time window. Set up alerts when success rates drop below acceptable thresholds. This monitoring catches problems early, whether they stem from proxy provider issues, target website changes, or problems in your scraping logic that manifest as proxy failures.
Each Python scraping library handles proxies differently, but the principles remain the same: configure the proxy address with authentication, implement rotation logic that tracks proxy health, and handle errors distinctly based on whether they originate from the proxy layer or the target website.