How to Crawl Complete Qidian Novels with Scrapy on Ubuntu

1. Overview

The purpose of this article is to use Scrapy to crawl completed novels from Qidian.com, using Ubuntu as the environment. Installation of Scrapy is assumed to be done separately.

2. Create Project

Run scrapy startproject name in the terminal, then navigate to the created project directory. Replace name with your desired project name.

3. Define Item

Define an item with fields such as title for the book name and desc for the book content.

4. Write Pipelines

In pipelines, specify how to store the scraped data; this example stores each book in a text file.

5. Write Settings

Replace the placeholder tutorial in the settings file with your project name.

6. Write Spider

Implement the spider to crawl the target pages. (The image shows the spider code.)

7. Conclusion

Although the code can retrieve all book contents, Qidian imposes VIP restrictions, requiring a VIP account to access completed novels, which limits this approach.