Why Your Python Web Scraper Fails and How to Fix It with Correct Headers

1. Introduction

A user asked about a Python web‑scraping problem where the request to the 10086 shop API returned unexpected results. The issue was traced to missing request parameters and insufficient HTTP headers.

2. Original Code

import requests

key = input("请输入关键字")
res = requests.post(
    url="https://jf.10086.cn/cmcc-web-shop/search/query",
    data={
        "sortColumn": "default",
        "sortType": "DESC",
        "pageSize": "60",
        "pageNum": "1",
        "firstKeyword": key,
        "integral": "",
        "province": ""
    },
    headers={"User_Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/113.0.0.0 Safari/537.36"}
)
print(res.json())

3. Solution

The data must be parsed as JSON, and the request needs additional headers, especially origin and a proper user-agent. Adding these headers resolves the issue.

import requests

headers = {
    "authority": "jf.10086.cn",
    "accept": "*/*",
    "accept-language": "zh-CN,zh;q=0.9,en;q=0.8",
    "content-type": "application/x-www-form-urlencoded;charset=UTF-8",
    "origin": "https://jf.10086.cn",
    "referer": "https://jf.10086.cn/",
    "user-agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/113.0.0.0 Safari/537.36 Edg/113.0.1774.42"
}
cookies = {
    "sajssdk_2015_cross_new_user": "1",
    "sensorsdata2015jssdkcross": "...",
    "BSFIT_EXPIRATION": "1684453169465",
    "BSFIT_DEVICEID": "..."
}
url = "https://jf.10086.cn/cmcc-web-shop/search/query"
data = {
    "sortColumn": "default",
    "sortType": "DESC",
    "pageSize": "60",
    "pageNum": "1",
    "firstKeyword": "食用油",
    "integral": "",
    "province": ""
}
response = requests.post(url, headers=headers, cookies=cookies, data=data)
print(response.json())

Running this corrected code returns the expected JSON data from the API.

4. Conclusion

The problem was caused by missing request headers; adding the proper User-Agent and Origin headers fixes the Python web‑scraping request and allows successful data retrieval.