How to Bypass Common Anti‑Scraping Mechanisms with Python

When learning web crawling, many sites employ anti‑scraping mechanisms that block direct data extraction. Understanding these defenses helps you devise solutions.

The two common mechanisms are identity verification and IP restrictions.

(1) Identity verification

Websites often identify crawlers through request headers, especially the User-Agent. A simple request to Douban’s Top 250 page returns no data because the default Python user‑agent is recognized as a bot.

import requests
url = 'https://movie.douban.com/top250'
res = requests.get(url)
print(res.text)

To mimic a real browser, capture the User-Agent from your browser’s developer tools (Network → first request → Request Headers) and include it in the request headers.

import requests
headers = {
    'user-agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_13_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/76.0.3809.132 Safari/537.36'
}
url = 'https://movie.douban.com/top250'
res = requests.get(url, headers=headers)
print(res.text)

The response now contains the full HTML of the page.

(2) IP restrictions

Websites may limit the number of requests from a single IP address. Excessive rapid requests trigger rate‑limiting, even if the User-Agent is spoofed. To avoid this, insert delays between requests using time.sleep().

import requests
import time
from bs4 import BeautifulSoup

def get_douban_movie(url):
    headers = {
        'user-agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/79.0.3945.130 Safari/537.36'
    }
    res = requests.get(url, headers=headers)
    soup = BeautifulSoup(res.text, 'html.parser')
    items = soup.find_all('div', class_='hd')
    for i in items:
        tag = i.find('a')
        name = tag.find(class_='title').text
        link = tag['href']
        print(name, link)

base_url = 'https://movie.douban.com/top250?start={}&filter='
urls = [base_url.format(num*25) for num in range(10)]
for item in urls:
    get_douban_movie(item)
    time.sleep(1)  # pause to avoid being blocked

The script fetches each page, extracts movie titles and links, and pauses one second between requests to stay under the site’s rate limit.

By adjusting request headers and throttling request frequency, you can successfully scrape data from sites that employ basic anti‑scraping defenses.