Scrape Complete Qidian Novels with Scrapy on Ubuntu – Step‑by‑Step Guide

Overview

The purpose of this article is to use Scrapy to crawl completed novels from the Qidian website, using an Ubuntu environment. Installation of Scrapy is assumed to be done separately.

Creating the Project

Run scrapy startproject name in the terminal, replacing name with your desired project name, to create the Scrapy project.

Defining Items

Define an Item with fields such as title for the book name and desc for the book content.

Writing Pipelines

In pipelines.py, implement storage logic; this example stores each book as a plain text file.

Settings

Replace the placeholder tutorial in the settings file with your project name.

Spider Implementation

Create a spider that navigates Qidian’s pages to fetch the novel titles and contents.

Conclusion

While the provided code can retrieve the content of completed novels, Qidian imposes a VIP restriction, requiring a logged‑in VIP account to access full novels; therefore the scraper cannot fetch VIP‑only books without such credentials.