Why Your Python Web Crawler Returns Wrong Data and How to Fix It

1. Introduction

Hello, I'm Pipi. In a Python community a member asked about a web‑crawler problem. The original script is shown below.

import requests
import json
import logging

logging.basicConfig(level=logging.INFO,
                    format='%(asctime)s - %(levelname)s: %(message)s...')

class Myspider():
    def __init__(self):
        self.url = "http://www.xinfadi.com.cn/getPriceData.html"
        self.headers = {
            "User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/104.0.0.0 Safari/537.36"
        }
        self.data = {
            "limit": 20,
            "current": '',
            "pubDateStartTime": "",
            "pubDateEndTime": "",
            "prodPcatid": "",
            "prodCatid": "",
            "prodName": ""
        }
        self.formdata = []

    def create_formdata(self):
        for i in range(1, 10):
            self.data['current'] = i
            self.formdata.append(self.data)
            # print(self.formdata)

    def get_html(self):
        for item in self.formdata:
            # print(item)
            html = requests.post(self.url, self.headers, item)
            html = json.loads(html.text)
            # print(html['current'])

    def run(self):
        self.create_formdata()
        self.get_html()

if __name__ == "__main__":
    spider = Myspider()
    spider.run()

The script runs without syntax errors, but the returned data is incorrect.

2. Solution

Teacher Yuliang provided a revised version of the code, shown below.

import requests
import json
import logging

logging.basicConfig(level=logging.INFO,
                    format='%(asctime)s - %(levelname)s: %(message)s...')

class Myspider():
    def __init__(self):
        self.url = "http://www.xinfadi.com.cn/getPriceData.html"
        self.headers = {
            "User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/104.0.0.0 Safari/537.36"
        }
        self.formdata = []
        for i in range(1, 10):
            self.data = {
                "limit": 20,
                "current": i,
                "pubDateStartTime": "",
                "pubDateEndTime": "",
                "prodPcatid": "",
                "prodCatid": "",
                "prodName": ""
            }
            self.formdata.append(self.data)

    def get_html(self):
        for item in self.formdata:
            # print(item)
            html = requests.post(self.url, headers=self.headers, data=item)
            print(html.json())
            # html = json.loads(html.text)
            # print(html)

    def run(self):
        self.get_html()

if __name__ == "__main__":
    spider = Myspider()
    spider.run()

Running this script produces the expected result.

3. Why the original code failed

The problem originates from reusing the same dictionary object when building the request payload. Each loop iteration modifies the same reference, so all entries end up with the last value. This is analogous to a=1, b=[a,a,a,a,a]; a=2; b.append(a), where all elements become 2.

4. Conclusion

The article identified a common mutable‑object pitfall in Python web crawlers and presented a corrected implementation that successfully fetches price data from the Xinfadi website.