Advanced Requests and Simple Web Scraping in Python: Extract Data in 5 Minutes

Core Goal

Connect the previous basic tutorial on requests with advanced techniques: set request headers, extract web data, and resolve mirror verification and parsing errors to complete a simple web‑scraping implementation.

1. Installing the Module (Avoid Mirror Errors)

USTC mirror requires browser verification and often fails for beginners; prefer Douban mirror (no verification) or Aliyun mirror as a backup.

Example commands:

pip install -i https://pypi.douban.com/simple requests
pip install -i https://mirrors.aliyun.com/pypi/simple/ requests

2. Core Review: Basic GET Request

import requests
url = "https://httpbin.org/get"
response = requests.get(url)
print(response.status_code)  # 200 = success

Advanced Requests Usage

1. Setting Request Headers (Impersonate a Browser)

Some sites block Python requests; adding a User-Agent header prevents interception.

import requests
headers = {
    "User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/124.0.0.0 Safari/537.36"
}
url = "https://httpbin.org/get"
response = requests.get(url, headers=headers, timeout=5)
print(response.status_code)
print(response.json())

2. Web Data Extraction (Simple Scraper)

Use response.text for raw HTML/text and combine string operations to pull key information, providing a fallback when JSON parsing fails.

import requests
headers = {"User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/124.0.0.0 Safari/537.36"}
url = "https://httpbin.org/get"
try:
    response = requests.get(url, headers=headers, timeout=5)
    response.raise_for_status()
    if "application/json" in response.headers.get("Content-Type", ""):
        data = response.json()
        print("Request URL:", data["url"])
        print("Client IP:", data["origin"])
    else:
        text = response.text
        ip_start = text.find('"origin": "') + len('"origin": "')
        ip_end = text.find('"', ip_start)
        ip = text[ip_start:ip_end]
        print("Client IP (text extraction):", ip)
except Exception as e:
    print(f"Request/parse error: {e}")
    if "网页解析失败" in str(e) or "不支持的网页类型" in str(e):
        print("Solution: check URL format, retry later, or use text extraction instead of JSON parsing")

3. Batch Requests (Extract Multiple Pages)

import requests
headers = {"User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/124.0.0.0 Safari/537.36"}
urls = [
    "https://httpbin.org/get?keyword=Python",
    "https://httpbin.org/get?keyword=爬虫",
    "https://httpbin.org/get?keyword=requests"
]
for url in urls:
    try:
        response = requests.get(url, headers=headers, timeout=5)
        response.raise_for_status()
        data = response.json()
        print(f"Keyword: {data['args']['keyword']}, URL: {data['url']}")
    except Exception as e:
        print(f"Request {url} failed: {e}")

4. Comprehensive Practice (Full Mini‑Crawler)

import requests
from datetime import datetime
headers = {"User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/124.0.0.0 Safari/537.36"}
urls = [
    "https://httpbin.org/get?keyword=Python入门",
    "https://httpbin.org/get?keyword=requests进阶",
    "https://httpbin.org/get?keyword=简易爬虫"
]
now_time = datetime.now().strftime("%Y-%m-%d %H:%M:%S")
print(f"Crawler start time: {now_time}")
for index, url in enumerate(urls, 1):
    try:
        response = requests.get(url, headers=headers, timeout=5)
        response.raise_for_status()
        data = response.json()
        keyword = data["args"]["keyword"]
        ip = data["origin"]
        print(f"#{index}: keyword={keyword}, client IP={ip}")
    except Exception as e:
        print(f"#{index} request failed: {e}")
        if "网页解析失败" in str(e):
            print("Tip: check URL format or retry later")

5. Common Pitfalls for Beginners

Pitfall 1: Installing from the USTC mirror triggers a browser‑verification error. Solution: Switch to Douban or Aliyun mirrors using the provided pip commands.

Pitfall 2: Requesting https://httpbin.org/get yields “网页解析失败”. Solution: Verify URL format, add a proper User-Agent, or fall back to text extraction.

Pitfall 3: 403 Forbidden due to request interception. Solution: Include the browser‑like header as shown in the advanced usage section.

Pitfall 4: Batch request stops on a single failure. Solution: Wrap each request in a try‑except block to keep the loop running.

6. Post‑Lesson Mini‑Task (5 Minutes)

# 1. Set headers, request the test API, extract client IP via JSON and via raw text
import requests
headers = {"User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/124.0.0.0 Safari/537.36"}
url = "https://httpbin.org/get"
try:
    r = requests.get(url, headers=headers, timeout=5)
    r.raise_for_status()
    print("JSON IP:", r.json()["origin"])
    text = r.text
    start = text.find('"origin": "') + len('"origin": "')
    end = text.find('"', start)
    print("Text IP:", text[start:end])
except Exception as e:
    print(f"Failed: {e}")

# 2. Batch request two URLs and print their keywords
urls = ["https://httpbin.org/get?keyword=数据提取", "https://httpbin.org/get?keyword=爬虫实操"]
for u in urls:
    r = requests.get(u, headers=headers, timeout=5)
    print("Keyword:", r.json()["args"]["keyword"])

Key Takeaways

Advanced requests focuses on header configuration and data extraction to overcome mirror verification and parsing errors.

Essential skills: setting User-Agent, handling JSON vs. raw‑text responses, performing batch requests, and robust exception handling.

When errors occur, first check URL format, then ensure headers are present, and finally switch extraction method.