Why Your Python Web Scraper Returns Wrong Data and How to Fix It

1. Introduction

Hello, I am PiPi. A few days ago a user asked a Python web‑scraping question in a chat group; the screenshot below shows the original issue.

The original code is shown below:

import requests
import json
import logging

logging.basicConfig(level=logging.INFO,
                      format='%(asctime)s - %(levelname)s: %(message)s...')

class Myspider():
    def __init__(self):
        self.url = "http://www.xinfadi.com.cn/getPriceData.html"
        self.headers = {
            "User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/104.0.0.0 Safari/537.36"
        }
        self.data = {
            "limit": 20,
            "current": 0,
            "pubDateStartTime": "",
            "pubDateEndTime": "",
            "prodPcatid": "",
            "prodCatid": "",
            "prodName": ""
        }
        self.formdata = []
    def create_formdata(self):
        for i in range(1,10):
            self.data['current'] = i
            self.formdata.append(self.data)
        # print(self.formdata)
    def get_html(self):
        for item in self.formdata:
            print(item) 
            html = requests.post(self.url,self.headers,item)
            html = json.loads(html.text)
            # print(html['current'])
    def run(self):
        self.create_formdata()
        self.get_html()

if __name__ == "__main__":
    spider = Myspider()
    spider.run()

The code looks syntactically correct, but it returns wrong data.

2. Implementation

A teacher provided a revised version of the script, shown in the image below.

The corrected code is:

import requests
import json
import logging

logging.basicConfig(level=logging.INFO,
                      format='%(asctime)s - %(levelname)s: %(message)s...')

class Myspider():
    def __init__(self):
        self.url = "http://www.xinfadi.com.cn/getPriceData.html"
        self.headers = {
            "User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/104.0.0.0 Safari/537.36"
        }
        self.formdata = []
        for i in range(1,10):
            self.data = {
                "limit": 20,
                "current": i,
                "pubDateStartTime": "",
                "pubDateEndTime": "",
                "prodPcatid": "",
                "prodCatid": "",
                "prodName": ""
            }
            self.formdata.append(self.data)
    def get_html(self):
        for item in self.formdata:
            # print(item)
            html = requests.post(self.url,headers=self.headers,data=item)
            print(html.json())
            # html = json.loads(html.text)
            # print(html)
    def run(self):
        self.get_html()

if __name__ == "__main__":
    spider = Myspider()
    spider.run()

Running this script produces the expected output:

Further questions revealed that the bug originated from reusing a mutable data structure. The example a=1, b=[a,a,a,a,a]; a=2; b.append(a) shows that after changing a, all elements in b become 2, leading to incorrect results.

3. Conclusion

The article presented a Python web‑scraping issue, analyzed the cause of the incorrect output, and provided a working implementation to help readers resolve similar problems.