Common Reasons Why Your Proxy Fails to Hide Your Web Scraper

In many developer communities, a frequent question is why a scraper that appears to use a proxy is still detected by target websites. This article outlines several common reasons that make proxy detection straightforward.

You Actually Didn’t Use a Proxy

Beginners often write code like the following, which only sets a proxy for HTTP URLs:

import requests

resp = requests.get('https://httpbin.org/ip').text
print('No proxy:', resp)

resp = requests.get('https://httpbin.org/ip', proxies={'http': 'http://IP:port'}).text
print('With proxy:', resp)

The output shows that the IP does not change because the proxy was not applied to the HTTPS request. To proxy HTTPS, both http and https keys must be set:

resp = requests.get('https://httpbin.org/ip', proxies={'http': 'http://IP:port', 'https': 'http://IP:port'}).text

P.S.: Note that the key for HTTPS proxies is https , but the value still starts with http:// . Using https:// as the value may cause errors with some proxy providers.

Your Proxy IP Is a Server IP

Many proxy providers purchase cloud servers (e.g., Alibaba Cloud, Tencent Cloud, Huawei Cloud, AWS, Google Cloud) to run proxy services. Cloud server IP ranges differ from residential IPs, and many websites block or challenge requests from these ranges.

Consequently, using such proxies often leads to detection, and only proxies based on residential IPs tend to evade blocks, albeit at higher cost.

Your Proxy Is Not a High‑Anonymity Proxy

Proxy types include transparent, anonymous, and high‑anonymity (elite) proxies. Transparent proxies reveal both the proxy and your real IP; anonymous proxies hide your real IP but still indicate proxy usage; only high‑anonymity proxies fully conceal proxy usage.

Free proxy lists frequently contain non‑elite proxies, which are easily detected.

Proxy IP Pool Is Polluted

Some scrapers are poorly written and generate noisy traffic, causing proxy IP pools to be quickly blacklisted. If many scrapers use the same provider and the provider’s pool isn’t refreshed promptly, the IPs become unreliable.

Proxy Does Not Support HTTP/2

Modern sites may require HTTP/2. While Python libraries like httpx support HTTP/2, most proxy services only handle HTTP/1.1, leading to failed requests when a client tries to use HTTP/2 through such a proxy.

Summary

Websites employ numerous methods to detect proxies, many of which can be implemented by a junior engineer without advanced techniques. Understanding these pitfalls helps avoid being blocked and prevents causing collateral damage to other scrapers.