Build a Car Model Scraper with Scrapy: Complete Step-by-Step Tutorial

Introduction

To obtain accurate and complete car model data for a work project, the author decided to scrape data from the well‑maintained automotive site Autohome using Python's Scrapy framework.

Preparation

Install Python 2.7.

Install Scrapy 1.4.0.

References

Autohome car‑model spider code (GitHub), Scrapy Chinese documentation, and an XPath tutorial are referenced for implementation details.

Project Initialization

Run the following command in the desired directory to create a new Scrapy project:

scrapy startproject myproject

When executing this step, an SSL error may occur: "TLSVersion.TLSv1_1: SSL.OP_NO_TLSv1_1" Fix it by installing a compatible Twisted version: sudo pip install twisted==13.1.0

Directory Structure

spiders/ – directory for spider logic.

items.py – defines data entities (Item classes).

middlewares.py – request/response middleware.

pipelines.py – processes items (cleaning, deduplication, etc.).

settings.py – configuration such as crawl speed, middleware activation, and output format.

Writing the Brand Spider

Create brand_spider.py in the spiders folder and define a BrandSpider class that inherits from scrapy.Spider. Set the name attribute (spider ID) and start_urls (entry URLs). Implement a parse method that extracts brand ID, URL, name, and icon URL using XPath expressions.

Define a corresponding BrandItem in items.py, inheriting from scrapy.Item, to hold the extracted fields. In parse, instantiate BrandItem, populate it with the extracted values, and yield the item so it flows through the pipeline.

Pipeline

The default pipeline in pipelines.py is left unchanged; it will receive the items and can perform cleaning or deduplication if needed.

Exporting to CSV

Modify settings.py to output results as CSV files:

FEED_FORMAT = 'csv'

FEED_URI = 'data/%(name)s_%(time)s.csv'

Running the Spider

Execute the spider from the project root: scrapy crawl brand The command generates a CSV file in the data directory containing the scraped brand data.

Rate Limiting and User‑Agent

To avoid being blocked by the target site, add a download delay and a custom User‑Agent: DOWNLOAD_DELAY = 3 Create user_agent_middlewares.py with a UserAgentMiddleware that randomly selects a User‑Agent string from a predefined list and injects it into each request.

Series and Model Spiders

Implement a series spider in spiders/series_spider.py similar to the brand spider. For model data, create spiders/model_spider.py using scrapy.spiders.CrawlSpider and Rule objects to follow pagination and detail links, because model pages are less regular.

Summary

The guide demonstrates a functional Autohome car‑model scraper built with Scrapy, covering essential components such as spiders, items, pipelines, CSV export, rate limiting, and user‑agent handling.

Tips

Use the Chrome XPath Helper extension (⌘+Shift+X on macOS) to test XPath expressions directly on the page.

Debug with scrapy shell : scrapy shell http://www.example.com then evaluate XPath with print(response.xpath('...')) .