Master Scrapy: Step‑by‑Step Guide to Crawl Beijing Xinfadi Price Data

Preface

Based on a previous theoretical article about Scrapy, this tutorial presents a hands‑on implementation for crawling Beijing Xinfadi price data.

Practical Exercise

Crawling Analysis

Open the Xinfadi price page in a browser and inspect the network requests. Each getPriceData.html returns JSON data. The request is a POST to http://www.xinfadi.com.cn/getPriceData.html with parameters limit (rows per page) and current (page number).

Example request body:

data={
    'limit': '20',
    'current': page
}

Alternatively, construct the URL directly:

http://www.xinfadi.com.cn/getPriceData.html?limit=20¤t=1
http://www.xinfadi.com.cn/getPriceData.html?limit=20¤t=2
http://www.xinfadi.com.cn/getPriceData.html?limit=20¤t=3

Create the Scrapy Project

Run the following command to start a new Scrapy project named Vegetables: scrapy startproject Vegetables The project structure is created as shown below:

Generate a spider with: scrapy genspider vegetables www.xinfadi.com.cn The generated vegetables.py contains:

import scrapy

class VegetablesSpider(scrapy.Spider):
    name = 'vegetables'
    allowed_domains = ['www.xinfadi.com.cn']
    start_urls = ['https://www.xinfadi.com.cn']

    def parse(self, response):
        pass

Define Items and Extract Data

In items.py define the fields to store:

import scrapy

class VegetablesItem(scrapy.Item):
    productName = scrapy.Field()
    lowPrice = scrapy.Field()
    highPrice = scrapy.Field()

Update the spider to request each page, parse the JSON response, and populate the item:

import scrapy
from Vegetables.items import VegetablesItem

class VegetablesSpider(scrapy.Spider):
    name = 'vegetables'
    allowed_domains = ['www.xinfadi.com.cn']

    def start_requests(self):
        for i in range(1, 3):
            url = f'http://www.xinfadi.com.cn/getPriceData.html?limit=20¤t={i}'
            yield scrapy.Request(url=url, callback=self.parse)

    def parse(self, response):
        html = response.json()
        fooddata = html.get('list')
        for i in fooddata:
            item = VegetablesItem()
            item['highPrice'] = i.get('highPrice')
            item['lowPrice'] = i.get('lowPrice')
            item['productName'] = i.get('prodName')
            yield item

Pipeline and Output

Add a pipeline entry in settings.py (the actual MongoDB pipeline is omitted for brevity):

ITEM_PIPELINES = {
    'Vegetables.pipelines.VegetablesPipeline': 300,
}

Run the spider and export results to CSV: scrapy crawl vegetables -o 11.csv Sample output screenshot:

Summary

Scrapy is an asynchronous, pure‑Python framework built on Twisted, ideal for extracting structured data with a clear architecture and low coupling between modules. This guide demonstrated creating a project, building a spider, handling pagination, extracting fields, and exporting data without storing it in a database.