Master Web Scraping in Python: From Basics to Bypassing Anti‑Scraping

Web scraping (爬虫) involves three essential elements: fetching, analysis, and storage.

Three Essential Elements of Web Scraping

Fetching a web page starts with resolving the domain to an IP address, establishing a TCP connection, and receiving the page content, which the browser then parses.

1. Basic Fetching

In Python 3.x, urllib is used for HTTP requests (the older urllib2 was removed). Example:

import urllib.request

response = urllib.request.urlopen('https://blog.csdn.net/weixin_43499626')
print(response.read().decode('utf-8'))

The requests library provides a more convenient interface:

import requests
# GET request
response = requests.get(url='https://blog.csdn.net/weixin_43499626')
print(response.text)
# GET with parameters
response = requests.get(url='https://blog.csdn.net/weixin_43499626', params={'key1':'value1','key2':'value2'})

2. Handling Login‑Required Pages

Submit a POST request with credentials, store the returned cookies, and reuse them for subsequent requests.

params = {'username':'root', 'passwd':'root'}
response = requests.post("http:xxx.com/login", data=params)
for key, value in response.cookies.items():
    print('key = ', key + ' ||| value :' + value)

Saving and loading cookies with urllib and http.cookiejar:

import urllib.request
import http.cookiejar

# Save cookies
cookie = http.cookiejar.MozillaCookieJar('cookie.txt')
handler = urllib.request.HTTPCookieProcessor(cookie)
opener = urllib.request.build_opener(handler)
request = urllib.request.Request('http://flights.ctrip.com/', headers={"Connection":"keep-alive"})
response = opener.open(request)
cookie.save(ignore_discard=True, ignore_expires=True)

# Load cookies
cookie = http.cookiejar.MozillaCookieJar()
cookie.load('cookie.txt', ignore_discard=True, ignore_expires=True)
handler = urllib.request.HTTPCookieProcessor(cookie)
opener = urllib.request.build_opener(handler)
request = urllib.request.Request('http://flights.ctrip.com/')
html = opener.open(request).read().decode('gbk')
print(html)

3. Dealing with Anti‑Scraping Measures

Set custom request headers (User‑Agent, Referer) to mimic a real browser:

headers = {
    'Host': 'https://blog.csdn.net',
    'Referer': 'https://blog.csdn.net/weixin_43499626/article/details/85875090',
    'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/68.0.3440.106 Safari/537.36'
}
response = requests.get("http://www.baidu.com", headers=headers)

Use IP proxies to avoid IP‑based blocking:

proxies = {
    "http": "http://119.101.125.56",
    "https": "http://119.101.125.1",
}
response = requests.get("http://www.baidu.com", proxies=random.choice([proxies]))

Introduce delays between requests to reduce request frequency:

import time
time.sleep(1)

For JavaScript‑rendered pages or complex interactions, Selenium can automate a real browser.

Analysis

After fetching, parse the HTML using regular expressions, BeautifulSoup , XPath, or lxml . XPath is generally faster because it is implemented in C.

Storage

Store the extracted data in plain text files, relational databases (MySQL), or NoSQL databases (MongoDB). Examples:

import json

dictObj = {
    '小明':{'age':15,'city':'beijing'},
    '汤姆':{'age':16,'city':'guangzhou'}
}
jsObj = json.dumps(dictObj, ensure_ascii=False)
with open('jsonFile.json','w') as f:
    f.write(jsObj)

import csv
with open('student.csv','w',newline='') as csvfile:
    writer = csv.writer(csvfile)
    writer.writerow(['姓名','年龄','城市'])
    writer.writerows([['小明',15,'北京'],['汤姆',16,'广州']])

import pymongo
client = pymongo.MongoClient('mongodb://127.0.0.1:27017/')
db = client.test
student_db = db.student
student_json = {'name':'小明','age':15,'city':'北京'}
student_db.insert(student_json)