Master Python Web Scraping: From Basics to Advanced Techniques

1. What is a web crawler

A web crawler (spider) is a program that sends requests to websites, retrieves resources, and extracts useful data.

2. Basic workflow

Two ways to obtain web data: (1) Browser request → download page → render; (2) Simulate browser request → extract data → store. Crawlers use method (2).

3. HTTP request and response

Requests are sent using an HTTP library; a response contains HTML, JSON, images, video, etc.

Key request components: method (GET/POST), URL, headers (User-Agent, Referrer, Cookie), and optional body.

Key response components: status codes (200 success, 301 redirect, 404 not found, 403 forbidden, 502 server error) and headers (e.g., Set-Cookie).

4. Essential Python modules

requests : simple HTTP library. GitHub: https://github.com/kennethreitz/requests.

re : regular expressions for text processing.

XPath : XML Path Language, used via lxml library.

BeautifulSoup : HTML/XML parser from bs4.

json : built‑in module for JSON handling.

threading : create threads by subclassing threading.Thread.

5. Example scripts

GET request example (demo_get.py): # demo_get.py content placeholder POST request example (demo_post.py): # demo_post.py content placeholder Proxy usage (demo_proxies.py) and AJAX data extraction (demo_ajax.py) are also shown.

Multithreaded crawling (demo_thread.py) demonstrates concurrent requests.

6. Scrapy framework

Scrapy is a Python‑based framework for extracting structured data from websites. It uses the asynchronous Twisted engine.

Core components: Scrapy Engine: coordinates spiders, pipelines, downloader, scheduler. Scheduler: queues and orders Request objects. Downloader: fetches responses. Spider: parses responses and yields items or new requests. Item Pipeline: processes extracted items (validation, storage). Downloader Middleware and Spider Middleware: allow custom processing of requests/responses.

Typical workflow: engine → scheduler → downloader → spider → pipeline → storage.

7. Building a Scrapy project

Create project: scrapy startproject mySpider Define items in items.py.

Generate spider: scrapy genspider gushi365 "gushi365.com" Implement pipelines in pipelines.py for storage.

8. Distributed crawling

scrapy‑redis extends Scrapy with Redis‑based components for distributed crawling.

Architecture: a Master node runs Redis for URL deduplication and request distribution; multiple Slave nodes run spiders that fetch URLs from Redis. pip install scrapy-redis GitHub: https://github.com/rolando/scrapy-redis

9. Useful tools

Fiddler – network capture tool for mobile.

XPath Helper – Chrome extension to locate XPath expressions.