Collection of Python Web Scraping Tools and Practical Examples

This document introduces a series of Python-based web‑scraping tools, each accompanied by a short description, installation instructions, and usage examples.

1. downloader.py – File Download Assistant

A simple utility for downloading images, videos, and files with progress display, easily integrable into other crawlers.

2. biqukan.py – Novel Downloader

Third‑party dependencies:

pip3 install beautifulsoup4

Usage:

python biqukan.py

3. video_downloader – VIP Video Downloader (iQIYI, etc.)

Source code folder: video_downloader

Install dependencies:

pip3 install -r requirements.txt

Run:

python movie_downloader.py

Supported platforms: Windows, Linux, macOS (Python 3).

4. baiduwenku.py – Baidu Wenku Article Scraper

Reference article: http://blog.csdn.net/c406495762/article/details/72331737 (note: code is for entertainment only).

5. shuaia.py – Image Scraper for "Shuaia" Website

Reference article: http://blog.csdn.net/c406495762/article/details/72597755

Install dependencies:

pip3 install requests beautifulsoup4

6. daili.py – Proxy IP Pool Builder

Reference article: http://blog.csdn.net/c406495762/article/details/72793480

7. carton – Scrapy Spider for "Naruto" Manga

Scrapes all chapters of the manga and saves them locally; the target site can be changed in settings.py .

Reference article: http://blog.csdn.net/c406495762/article/details/72858983

8. hero.py – "Honor of Kings" Equipment Recommendation Helper

Demonstrates extending web scraping to mobile app data.

Reference article: http://blog.csdn.net/c406495762/article/details/76850843

9. financical.py – Financial Report Downloader

Shows how to store scraped data into a database; see related article for details.

Reference article: http://blog.csdn.net/c406495762/article/details/77801899

10. one_hour_spider – One‑Hour Introduction to Python3 Web Crawling

Covers novel download, wallpaper download, and iQIYI video download.

References:

Zhihu: https://zhuanlan.zhihu.com/p/29809609

CSDN: http://blog.csdn.net/c406495762/article/details/78123502

11‑13. douyin.py / douyin_pro / douyin_pro_2 – Douyin (TikTok) Video Downloaders

Various versions add watermark removal and third‑party URL parsing.

Reference article: http://cuijiahua.com/blog/2018/03/spider-5.html

14. geetest.py – GEETEST CAPTCHA Bypass

Explains how to defeat sliding CAPTCHAs provided by Geetest.

Reference article: http://www.cuijiahua.com/blog/2017/11/spider_2_geetest.html

15. 12306.py – Simple Train Ticket Snatching Script

Basic script for automating ticket purchase on 12306.

16. baiwan.py – "Million Hero" Quiz Assistant

Uses Python to fetch quiz data, match answers via Baidu Zhidao, and push results to a web client.

17. Netease – NetEase Cloud Music Downloader

Downloads songs based on a playlist file (music_list.txt).

18. bilibili.py – Bilibili Video and Danmaku Batch Downloader

Usage example:

python bilibili.py -d 猫 -k 猫 -p 10

Parameters:

-d: output folder name

-k: search keyword

-p: number of result pages to download

Full source code repository: https://github.com/Jack-Cherish/python-spider

Additional resources and recommended reading links are provided at the end of the article.