Scrapy Tutorial: Installation, Components, Project Setup, Code Implementation, and Data Storage

Scrapy is a fast, high‑level Python framework for web crawling and data extraction. The tutorial begins with installing the required dependencies, including Twisted and Scrapy itself, using pip install scrapy.

It then explains Scrapy's architecture, listing its main components: Engine, Scheduler, Downloader, Spider, Item Pipeline, Downloader Middleware, Spider Middleware, and Scheduler Middleware, and describes the typical data flow among them.

The project analysis targets the AQI history data site (https://www.aqistudy.cn/historydata/). A new Scrapy project is created with scrapy startproject weather_spider, and a spider named weather is generated using scrapy genspider weather www.aqistudy.cn/historydata.

In items.py an WeatherSpiderItem is defined with fields for city, date, AQI, level, PM2.5, PM10, SO2, CO, NO2, and O3_8h. The settings.py file is updated with a list of user‑agent strings stored in MY_USER_AGENT and the middleware activation order (RandomUserAgentMiddleware at priority 900).

A RandomUserAgentMiddleware class selects a random user‑agent for each request. To handle pages rendered by JavaScript, a WeatherSpiderDownloaderMiddleware is implemented; when a request contains meta['selenium'] = True, it launches a headless Chrome instance via Selenium, retrieves the rendered HTML, and returns a scrapy.http.HtmlResponse.

The spider code includes three parsing methods: parse: extracts city URLs and names from the start page and yields requests to parse_month. parse_month: extracts the first five month URLs for a city, passes the city name and a selenium flag, and yields requests to parse_day_data. parse_day_data: iterates over table rows, populates a WeatherSpiderItem with daily AQI data, and yields the item.

Running the project from the project root can be verified with scrapy list. The crawler can export data in JSON, JSON Lines, CSV, or XML formats, e.g., scrapy crawl weather -o spider.json. To ensure correct encoding, FEED_EXPORT_ENCODING = 'utf-8' is added to settings.py.

For persistent storage, a MongoDB pipeline is defined in pipelines.py. The pipeline connects to MongoDB using MONGO_URI and MONGO_DB settings, inserts each item into a collection named after the item class, and closes the connection when the spider finishes. The pipeline is activated via

ITEM_PIPELINES = {'weather_spider.pipelines.MongoPipeline': 300}

.

The article concludes that the presented Scrapy setup can be extended to crawl all cities' weather data from the target site, demonstrating a complete end‑to‑end workflow from installation to data persistence.