How to Build a Scrapy Spider to Crawl AutoHome Car Data in Python

Preparation

Install Python 2.7 and the Scrapy module (version 1.4.0).

References

Code repository: AutoHome spider . Scrapy Chinese documentation and an XPath tutorial are also recommended.

Initialize Project

Run the following command in the desired directory: scrapy startproject 项目名称 If an exception occurs: "TLSVersion.TLSv1_1: SSL.OP_NO_TLSv1_1" Resolve it by installing a compatible Twisted version:

sudo pip install twisted==13.1.0

Directory Structure

spiders : directory for spider logic.

items.py : defines data entity classes.

middlewares.py : request/response middleware.

pipelines.py : data processing pipeline.

settings.py : configuration file (crawl speed, middleware order, output format, etc.).

Define Target Data

The goal is to collect car brands, series, and models. Brand data is loaded lazily on the Autohome brand list page; each alphabet letter (A‑Z) loads a separate URL (e.g., http://www.autohome.com.cn/grade/carhtml/B.html), allowing full brand extraction.

Write the Brand Spider

Create brand_spider.py in the spiders directory and define a BrandSpider class inheriting from scrapy.Spider. Set the name and start_urls, then implement a parse method that extracts brand ID, URL, name, and icon using XPath and yields BrandItem instances.

In items.py , define a BrandItem class inheriting from scrapy.Item with fields for the extracted brand data.

Pipeline

The default pipeline.py processes items (e.g., cleaning, deduplication). No changes are required for this simple spider.

Export to CSV

Modify settings.py to set the output format and file name:

FEED_FORMAT = 'csv'

FEED_URI = 'data/%(name)s_%(time)s.csv'

Run the Spider

Execute the spider from the project root: scrapy crawl brand A CSV file with brand data will appear in the data directory.

Avoid Being Blocked

To prevent the target site from blocking high‑frequency requests, add a download delay: DOWNLOAD_DELAY = 3 Also set a random User‑Agent header by creating user_agent_middlewares.py with a UserAgentMiddleware that selects a user‑agent from a predefined list.

Series and Model Spiders

Series spider is similar to the brand spider (implemented in spiders/series_spider.py ). The model spider is more complex, using CrawlSpider and Rules to follow additional URLs and parse irregular pages.

Tips

Use the Chrome XPath Helper extension (⌘+Shift+X) to test XPath expressions directly on the page.

Debug with scrapy shell : scrapy shell http://www.xxxxx.xx Then evaluate XPath with print response.xpath('...') .