How to Set Up Rotating Proxies in Scrapy

Steps

1. Keep HttpProxyMiddleware enabled (it is on by default)
2. Set meta={"proxy": url} on each request, or in start_requests
3. Put the username and password in the proxy URL
4. Lower CONCURRENT_REQUESTS_PER_DOMAIN if a site still rate limits

Example code

import scrapy

PROXY = "http://USERNAME-cc-gb:PASSWORD@gw.gbproxies.cc:10000"

class PricesSpider(scrapy.Spider):
    name = "prices"
    start_urls = ["https://example.com/catalog"]

    def start_requests(self):
        for url in self.start_urls:
            yield scrapy.Request(url, meta={"proxy": PROXY})

    def parse(self, response):
        for href in response.css("a.product::attr(href)").getall():
            yield response.follow(href, self.parse_item, meta={"proxy": PROXY})

    def parse_item(self, response):
        yield {"title": response.css("h1::text").get(), "url": response.url}

Replace USERNAME and PASSWORD with the proxy login from the Proxies page of the console. Port 10000 is residential HTTP; see the ports of every product below.

Tips

• Set IMAGES_ENABLED off and avoid media downloads to save GB
• Use DOWNLOAD_TIMEOUT 30 and RETRY_TIMES 3 for steady runs