Bypass Anti‑Scraping Measures with Python Requests and Proxy Pools

Introduction

In a previous article we showed how to scrape free proxy IPs and verify their availability using Python. This series will cover proxy site overview, anti‑scraping techniques, data extraction, and visualization. This first part focuses on the proxy site and its anti‑scraping measures.

Proxy Site Overview

The target website aggregates tens of thousands of proxy IPs, both free and paid. Free proxies are often unreliable and may stop working shortly after being listed.

Anti‑Scraping Measures

The site employs several defenses that block simple requests:

Requests without any header receive no data.

Repeated access from the same IP (over 40 times) results in the IP being blocked.

To address these issues we use two main strategies:

Capture normal browser http request headers using a traffic‑sniffing tool (Fiddler) and include them in requests calls, making the traffic appear as if it originates from a real browser.

Build a rotating proxy pool: obtain an initial list of proxies from other sources, continuously add newly discovered proxies, randomly select a proxy for each request, and promptly remove any proxy that becomes invalid or blocked.

We captured the required headers with Fiddler, as shown below:

After extracting the headers, we wrap them into a dictionary that requests can use. The following function returns the prepared header dictionary:

When making a request, we simply pass this header dictionary to requests:

Conclusion

We have prepared the necessary anti‑scraping measures, including proper request headers and a rotating proxy pool. The next article will dive into page structure analysis and data extraction techniques.