How to Bypass Anti‑Scraping Measures: User‑Agent, Cookies & Proxies

When writing web crawlers, many sites implement anti‑scraping measures that can quickly block your IP, especially when scraping large volumes of data.

The article summarizes several countermeasures that can be applied individually or together for better results.

Fake User‑Agent

Set the User-Agent header to mimic a real browser.

Example:

headers = {'User-Agent': 'Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/52.0.2743.116 Safari/537.36'}
resp = requests.get(url, headers=headers)

You can also collect multiple browser User‑Agents and randomly select one for each request to increase anonymity.

Random Delays

Introduce a random pause between requests to avoid rapid request patterns.

time.sleep(random.randint(0, 3))  # pause 0‑3 seconds
# or
time.sleep(random.random())          # pause 0‑1 second

Fake Cookies

If a page can be accessed in a browser, copy its cookies and use them in your requests.

Note: Even with cookies, excessive request frequency may still trigger IP bans; manual verification (e.g., captcha) may be required.

Use Proxies

Rotate multiple proxy IPs to distribute requests and prevent a single IP from being blocked.

Anti‑Anti‑Spider Project

For advanced counter‑measures, refer to the "Anti‑Anti‑Spider" project on GitHub, which collects various techniques to evade anti‑scraping defenses.

Link: github.com/luyishisi/Anti-Anti-Spider