This article introduces web crawling fundamentals, demonstrates how to inspect HTML elements, walks through simple examples using urllib, requests, and BeautifulSoup, and culminates in a complete Python script that extracts chapter links and contents from a novel website, saving them to a text file.
This article provides a comprehensive tutorial on configuring front‑end resource bundling, adjusting server settings, building and uploading SpringBoot back‑end modules, creating deployment scripts, managing environment‑specific properties, and implementing a Baidu Tieba hot‑search crawler, enabling a complete end‑to‑end cloud deployment.
This article explains why a Python web crawler often fails to retrieve JavaScript‑rendered page content, outlines common causes such as dynamic loading, anti‑scraping measures, and server‑side rendering, and offers practical techniques to capture the fully rendered HTML.
Learn the core concepts of using proxies in Python web scraping, including what proxies are, common types like anonymous and high‑anonymity, how they protect your crawler, practical implementation with the requests library, and an overview of building a proxy pool for scalable data extraction.
This article presents a curated list of Python web‑scraping utilities—including file download assistants, novel and video grabbers, proxy pool builders, and various automation scripts—along with installation commands, usage examples, source links, and brief operational explanations for each tool.
This tutorial explains how to set up the PHP environment, install the Goutte library, and use it to fetch page content, extract hyperlinks, and submit forms, providing complete code examples for building a functional web crawler.
This article walks through a Python web‑scraping solution for extracting and downloading audio files from a WeChat page, explains the problem, shows the implementation steps with screenshots, and provides the complete, ready‑to‑run code.
This article walks through using the free online tool spidertools.cn to automatically convert captured HTTP requests into ready‑to‑run Python requests code for web scraping, showing step‑by‑step screenshots and explaining how the method works for common GET and POST scenarios.
This article explains what a web crawler is, describes the basic environment and tools needed for Python crawling, outlines the typical scraping workflow, and presents three implementation styles—basic, function‑encapsulated, and concurrent—illustrated with diagrams and practical guidance.
Photon is a fast, multithreaded Python web crawler that extracts URLs, files, and intelligence such as emails and social media accounts, offering flexible options, Ninja mode, and extensive command‑line parameters for precise and efficient data harvesting across multiple operating systems.
This article walks you through using Scrapy to fetch price data from Beijing Xinfadi's website, covering request analysis, spider creation, item definition, pagination, data extraction, pipeline setup, and exporting results to CSV with full code examples.
This article explains how to create a Python web crawler that automatically generates URLs for a drama‑download site, filters out invalid pages, extracts ed2k links using requests and regular expressions, saves them to text files, and employs multithreading to speed up processing, while discussing challenges such as duplicate URLs and filename sanitization.
This article provides a comprehensive, step‑by‑step guide to the Scrapy web‑crawling framework, covering its core components, installation methods, project layout, spider creation, data extraction techniques, pagination handling, pipeline configuration, and how to run the crawler to collect and store data.
This comprehensive guide introduces Scrapy's architecture, explains its core components and data flow, teaches XPath fundamentals, walks through installation, project creation, spider coding, item and pipeline definitions, middleware customization, pagination handling, and essential settings for effective Python web crawling.
This tutorial walks through analyzing a Qidian novel page, extracting GET request parameters, constructing the catalog URL, and repeatedly fetching free chapters with Python, handling asynchronous loading and delays to reliably download the novel’s free content.
This article walks through building a Python web crawler that extracts forum post comments into MongoDB, covering project goals, environment setup, site structure analysis, Scrapy and Selenium integration, data storage design, handling anti‑scraping measures, and performance optimization with multithreading.
This tutorial walks you through installing Scrapy on Windows, creating a project and spider, configuring settings, and using XPath to crawl and extract joke titles and contents from the Qiushibaike website, providing a solid foundation for Python web scraping.
This tutorial shows how to locate the "next page" link on a website, extract its URL using Scrapy selectors, add proper checks, and integrate the pagination logic into a Scrapy spider so that all article pages are crawled automatically.
This article provides a comprehensive step‑by‑step guide to installing Scrapy, understanding its core components and processing flow, creating a weather‑data crawling project, writing items, settings, middlewares, spiders, running the crawler, exporting results, and storing the scraped data into MongoDB.
This guide introduces Scrapy, a powerful Python web‑scraping framework, explains its architecture and components, walks through installation, project creation, spider development, query syntax, recursive crawling, and item pipelines, providing practical code examples for building robust crawlers.
This article describes how the author built a Python web crawler to automatically generate numeric URLs, fetch TV drama pages from the 天天美剧 site, extract ed2k download links using regular expressions, and save them into organized text files, streamlining the download process with Thunder.
This tutorial walks you through creating a Scrapy project, defining items, writing a spider, and exporting scraped data to JSON while covering common pitfalls like encoding errors and XPath selector usage for extracting titles, URLs, and descriptions from dmoz.org.
This guide introduces Scrapy, a fast Python web‑crawling framework, explains its architecture, installation, project setup, spider creation, execution, and advanced features like XPath selectors, recursion, and item pipelines, providing a complete hands‑on tutorial.
This article presents a complete PHP configuration file for a flexible web crawler, detailing database connection parameters, target site URLs, pagination controls, content extraction patterns, and optional text filters to help developers quickly set up and customize their scraping projects.
This article explains how to create a Python web‑scraper that automatically generates URLs, fetches drama pages from a download site, extracts ed2k links with regular expressions, saves them to text files, and handles missing pages and filename restrictions efficiently.
This tutorial walks beginners through creating a minimal Scrapy project, writing a spider that fetches forum thread titles and content, extracting data with XPath, and extending the crawler with pipelines, middleware, and common settings for robust web scraping.
This guide walks through building a Python web crawler to extract novel titles and URLs from the qu.la ranking page, explains the site’s clear HTML structure, shows how to deduplicate entries with a set, and provides complete code snippets plus performance tips and a Scrapy upgrade path.
This article presents a curated list of 32 Python web‑scraping projects, each with a brief description of its purpose, technology stack, and data output format, helping developers quickly find useful open‑source crawlers on GitHub.
This guide walks through the complete process of crawling the Sichuan University Public Administration College website to extract every news article, covering target identification, rule definition, code implementation, handling pagination, and troubleshooting missing items.
This article explains how to create a Python web crawler for Baidu Tieba that extracts the original poster's content, determines page counts, retrieves the thread title, and saves all posts into a local TXT file, complete with usage instructions and code details.
This tutorial walks you through creating a Python-based web crawler that fetches jokes from Qiushibaike, explaining the page structure, required regular expressions, and how to run the script to browse content directly from the command line.
