Multiple Anti‑Crawling Measures and Best Practices for Web Scraping

1. IP Restriction – Detect unusually frequent requests from a single IP and temporarily block that IP to prevent automated scraping.

2. User‑Agent Validation – Identify the browser or client via its User‑Agent string; legitimate browsers have known patterns, while unknown or suspicious agents can be blocked. Example User‑Agent strings include:

<code>User-Agent: Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10_6_8; en-us) AppleWebKit/534.50 (KHTML, like Gecko) Version/5.1 Safari/534.50</code>

<code>User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.6; rv:2.0.1) Gecko/20100101 Firefox/4.0.1</code>

<code>User-Agent: Mozilla/5.0 (Windows NT 6.1; rv:2.0.1) Gecko/20100101 Firefox/4.0.1</code>

Legitimate User‑Agents vary across browsers and operating systems; any request lacking a normal User‑Agent is likely a crawler and can be restricted.

3. CAPTCHA Verification – Deploy CAPTCHAs to block automated programs; modern CAPTCHAs use complex noise and distortion that are difficult for bots to solve.

4. AJAX Asynchronous Loading – Load content dynamically via JavaScript, making it harder for simple crawlers that do not execute JS to retrieve data.

5. <noscript> Tag Usage – Serve alternative content when JavaScript is disabled; many low‑level crawlers lack JS support, so combining <noscript> with AJAX can protect sensitive information.

6. Cookie Restrictions – Monitor cookie presence across multiple visits; if a client repeatedly lacks expected cookies, it may be a crawler and can be blocked.

Crawler Development Tips

Respect ethical considerations and obey the robots.txt protocol.

Avoid infinite loops by parsing URLs carefully with tools like urlparse .

Set reasonable request timeouts (e.g., shorter than the default 200 seconds) to prevent thread bottlenecks.

Implement efficient duplicate‑detection to avoid revisiting the same pages.

Consider a download‑first, analysis‑later workflow to speed up crawling.

Handle resource deadlocks in asynchronous code.

Use precise element selectors (e.g., XPath) to reduce dirty data.

Images illustrating User‑Agent examples and CAPTCHA samples are included in the original article.

Disclaimer: The content is compiled from online sources; copyright belongs to the original author. Please contact us for removal or authorization if any rights are infringed.