Python Basics – Part 8: Master requests in 5 Minutes to Start Web Scraping

Prerequisite Review

# Import the module (choose one)
import requests  # most common
# from requests import get, post  # import specific functions
# import requests as req  # alias for brevity

Core usage of requests

Simple GET request

import requests

url = "https://httpbin.org/get"  # stable test endpoint
response = requests.get(url)  # send request, get response

print(response.status_code)   # 200 = success, 404 = not found
print(response.text)          # raw HTML/text
print(response.json())       # JSON data (auto‑parsed for APIs)

GET request with parameters (simulate a search)

import requests

url = "https://httpbin.org/get"
params = {
    "keyword": "Python",   # search keyword
    "page": 1                # page number
}

response = requests.get(url, params=params)
print("Full request URL:", response.url)   # verify parameters were added
print(response.json())                     # JSON response with echoed params

Exception handling (avoid crashes)

import requests

url = "https://httpbin.org/get"
try:
    response = requests.get(url, timeout=5)  # set timeout to prevent hanging
    response.raise_for_status()            # raise if status is not 200
    print("Request succeeded, response:")
    print(response.json())
except Exception as e:
    print(f"Request failed: {e}")
    if "解析失败" in str(e) or "不支持的网页类型" in str(e):
        print("Solution: check the URL or switch to another endpoint, then retry")

Integrated practice (requests + built‑in modules)

# Real‑world example: fetch data, extract key info, record timestamp
import requests
from datetime import datetime

url = "https://httpbin.org/get"
params = {"keyword": "Python入门", "page": 1}

try:
    response = requests.get(url, params=params, timeout=5)
    response.raise_for_status()
    data = response.json()  # parse JSON

    request_url = data["url"]
    args = data["args"]
    now_time = datetime.now().strftime("%Y-%m-%d %H:%M:%S")

    print(f"Request time: {now_time}")
    print(f"Request URL: {request_url}")
    print(f"Request params: {args}")
except Exception as e:
    print(f"Request failed: {e}")
    if "解析失败" in str(e):
        print("Please check the URL or retry later")

Common pitfalls for beginners

Pitfall 1: ModuleNotFoundError when importing requests – the module is not installed. Solution: reinstall using a reliable mirror (e.g., the USTC mirror).

Pitfall 2: Connection refused – wrong or inaccessible URL. Solution: use the stable test endpoint https://httpbin.org/get instead of unavailable sites.

Pitfall 3: JSONDecodeError – response is not JSON. Solution: print response.text first; if it’s not JSON, process the raw text accordingly.

Pitfall 4: Request timeout or parsing failure. Solution: add a timeout argument (e.g., timeout=5) and wrap the call in try‑except as shown above.

Post‑lesson task (complete in 5 minutes)

# 1. Install requests via USTC mirror
# pip install -i https://pypi.mirrors.ustc.edu.cn/simple/ requests

# 2. Send a GET request and print status code and JSON response
import requests
url = "https://httpbin.org/get"
response = requests.get(url)
print("Status code:", response.status_code)
print("JSON response:", response.json())

# 3. Add parameters (keyword="爬虫", page=2) and print the full URL
params = {"keyword": "爬虫", "page": 2}
response = requests.get(url, params=params)
print("Full request URL:", response.url)

Key takeaways

Requests is the foundation for crawling and API calls; master GET requests, parameter passing, and exception handling.

Essential steps: install the library (avoid mirror errors), send GET requests, parse responses, and catch exceptions.

When errors occur, first check module installation, mirror configuration, URL validity, and data format compatibility.