This article explains the legal boundaries of web crawling in China, covering recent cybersecurity regulations, what makes a crawler illegal or legal, common developer questions, and practical advice to avoid personal‑data violations and criminal liability.
This article explains how to scrape Chinese Basketball Association (CBA) match data from a portal website, analyzes the page structure, extracts table rows using regular expressions, converts them to CSV format, and provides a complete Java/Groovy code example for automated data collection.
This article presents seven practical Python scripts—from scraping Zhihu images and chatting bots to poetry author detection, lottery generation, auto‑drafting apologies, screen recording, and GIF creation—showcasing how to quickly automate diverse tasks without reinventing the wheel.
This article explains how to use a Groovy script to crawl over 7 million historical weather entries for 3,200 cities spanning 2011‑2019, process the JSON responses, and store the cleaned data into a MySQL table, while sharing practical tips and code snippets.
This guide explains practical techniques such as faking User‑Agent headers, rotating cookies, adding random delays, and using proxy pools to prevent IP bans while crawling large amounts of data from websites with anti‑scraping defenses.
This guide walks through using Python's requests and BeautifulSoup libraries to simulate Zhihu login, extract user profiles, retrieve answer likers, download avatars, fetch all answers for a question, and store the collected data in a SQLite database.
This tutorial walks you through using Python to collect tourism data from Qunar, extract key fields such as name, price, and rating, store the results in Excel with pandas, and visualize sales and popularity trends using pyecharts, including a simple recommendation algorithm.
This article introduces Puppeteer, a Node library for controlling Chrome, and demonstrates its use in automating web performance analysis and scraping tasks.
This article demonstrates how to use Selenium and Python to crawl over 6,000 Chinese tourist attractions from Qunar, extract ratings, popularity and sales data, and visualize the results with pandas, seaborn, matplotlib, and pyecharts, revealing the most visited sites and regional travel trends during the 2019 National Day holiday.
This article walks through the challenges of scraping Dazhong Dianping, explains how the site hides numeric data with custom CSS fonts, and provides a complete Python workflow—including HTTP requests, font extraction, glyph rendering, and OCR—to decode and retrieve the protected information.
This guide explains core XPath concepts—including node selection shortcuts, predicate filters, wildcard usage, multiple path unions, axis navigation, and built‑in functions—providing clear examples for effective XML/HTML data extraction.
After uncovering an unprotected API that allowed unlimited resource access, the author created a rough Java program that uses a fixed-size thread pool and CountDownLatch to fetch 100 000 items in parallel, retrieving 10 000 records per thread via HTTP GET requests.
The article walks through a Java solution for extracting multilingual data from an HTML table using regular expressions, handling spacing and encoding issues, splitting fields, and constructing INSERT statements to populate a country_code database table.
This tutorial walks you through logging into Taobao with Python requests, handling anti‑scraping measures, extracting product information via the PC search API, parsing JSON data, and saving the results to Excel, while also covering common pitfalls like sliders and proxy management.
Learn a step‑by‑step Python solution to locate, download, and store high‑resolution, royalty‑free images from ColorHub by navigating its three‑tier page structure, generating request headers, parsing HTML with BeautifulSoup, and saving files locally, enabling offline PPT creation without copyright concerns.
This tutorial introduces BeautifulSoup, a powerful Python library for parsing HTML and XML, covering installation, basic usage, tag selection, attribute extraction, navigation of parent and sibling nodes, method and CSS selectors, and best‑practice recommendations for efficient web data extraction.
This tutorial walks you through creating a Python‑based GitHub monitoring tool that logs in, crawls code search results for sensitive keywords, extracts repository details, writes findings to CSV, and sends email alerts, providing a practical approach to detecting accidental source‑code leaks.
The article explains how to build a poster‑rendering service with Puppeteer, detailing its advantages over canvas, the Redis‑based caching and CDN workflow, optimization tricks for headless Chromium, and future plans to boost QPS and pre‑generate popular posters.
This article walks through a complete data‑analysis workflow—scraping Zhihu dating‑preference answers, cleaning and filtering the data, deriving gender and activity metrics, designing a four‑step screening process, and finally ranking candidates with a custom like‑to‑comment index—to help a single programmer create a concise, high‑quality list of potential partners.
This article walks through collecting Python job postings from Lagou by handling anti‑scraping measures, parsing POST requests, storing results in Excel, and then using pyecharts to create bar, map, and pie visualizations that reveal city distribution, salary ranges, and experience requirements.
The article explains how to create a WeChat Mini Program backend with Tencent Cloud development, use the React‑based Taro framework to build a Japanese sentence helper, consolidate multiple cloud functions via tcb-router, and scrape example sentences with superagent and cheerio, highlighting setup tips and known limitations.
This tutorial explains how to extract Meituan Takeout app comments by analyzing Ajax requests, constructing dynamic URLs, parsing JSON with regular expressions, and storing the results in text files and MongoDB using Python.
This guide explains why Scrapy on Windows raises the "No module named win32api" error, walks through installing the correct pywin32 package (or pypiwin32), shows how to obtain the proper wheel from an unofficial source, and provides extra tips for locating Scrapy spider names.
This article details a complete workflow for extracting nationwide tourist attraction data from Qunar, cleaning and enriching it with geographic coordinates, and performing multi‑level statistical analysis and visualizations—including sales rankings, popularity metrics, heatmaps, and word clouds—to reveal regional tourism patterns across China.
This tutorial demonstrates how to use asynchronous Python (aiohttp, motor, asyncio) to scrape all comments from a popular Bilibili video, store them in MongoDB, and generate a Chinese word‑cloud analysis to reveal why the video went viral.
This article walks through building a Python Scrapy spider to extract comprehensive car brand, series, and model data from Autohome, covering environment setup, project initialization, spider and item definitions, handling lazy-loaded pages, CSV output configuration, rate limiting, user‑agent rotation, and debugging tips.
This guide explains the core XPath syntax—including node selectors, predicates, wildcards, multiple paths, axes, and built‑in functions—providing clear examples so you can efficiently locate and filter XML/HTML elements when scraping web pages.
Using Python’s Scrapy, pandas, and seaborn, the author scraped and cleaned public Shanbay user data, stored it in PostgreSQL, and analyzed registration and study habits to reveal that over 68% of users abandon word‑learning on day one, with only a tiny fraction persisting beyond 100 days.
This article shows how to use Python to scrape Zhihu answers, store them in MongoDB, filter for short high‑upvote replies, and then presents a collection of programmer‑centric jokes that illustrate the kind of "god replies" the crawler can retrieve.
This guide explains how to extract novel titles and links from a structured ranking website, remove duplicate entries using a set, handle HTML tags, and improve crawling speed with multithreading or the Scrapy framework, all while keeping the code modular and reusable.
This guide explains how to scale web crawling to hundreds of sites using Scrapy‑Redis, covering its components, distributed workflow, Redis installation and configuration, proxy pool handling, and provides complete Python code examples for spiders and pipelines.
This article explores headless browser automation technologies including Selenium, PhantomJS, Puppeteer, and Headless Chrome, comparing their architectures, use cases, and implementation differences.
This article describes how to use Python, Selenium, and search‑index services to scrape and visualize popularity data for Chinese tourist spots during the National Day holiday, presenting a ranking of destinations and providing full code examples for data collection, cleaning, and storage.
This article explains why regular expressions are essential for Python web scraping, introduces the special characters ^, ., and *, and demonstrates their use with clear code examples, showing how to extract specific patterns such as numbers from HTML content.
This tutorial walks through using Python and browser developer tools to locate the NetEase Cloud Music comment API, extract encrypted request parameters, retrieve hot comments in JSON, and parse the Hot Songs chart to collect each song's name and ID with regular expressions.
This article provides a comprehensive Python web scraping guide covering basic page retrieval with GET and POST, proxy usage, cookie handling, header spoofing, page parsing techniques, captcha processing, gzip compression, and multithreaded crawling, complete with code snippets for each step.
This tutorial shows how to export WeChat Moments using a third‑party service, then build a Python Scrapy spider to crawl the exported pages, parse the JSON data, and save the moments to a file, with detailed commands and code examples.
Using Python to scrape over ten thousand Maoyan comments for the comedy film “The Billionaire” (西虹市首富), this article demonstrates data cleaning, geographic heat‑maps, city‑wise rating analysis, word‑cloud generation, and a simple box‑office forecast based on a comparable movie, illustrating practical web‑scraping and data‑mining techniques.
This article introduces Beautiful Soup, a Python library for parsing HTML/XML into a navigable tree, covering installation, object initialization, tag and attribute access, tree traversal, searching techniques like find_all, find, CSS selectors, and practical code examples.
The article walks through a full Python workflow that scrapes up to 500 Douban movie reviews for "Dying to Survive" and "Hidden Blade," cleans and stores them in pandas, performs SnowNLP sentiment analysis, and visualizes city distribution, rating trends, and word clouds with pyecharts.
Learn to automate course enrollment on outdated university portals by simulating login with Python, extracting session data, handling CAPTCHAs, constructing POST requests, and parsing responses, enabling fast, reliable class registration even when schedules conflict.
This tutorial walks you through obtaining WeChat Moments data via a third‑party export service, setting up a Scrapy project, analyzing the JSON responses, and implementing the spider code to extract and save posts and timestamps.
This guide demonstrates how to programmatically log into the Super Schedule mobile app using Python, capture its JSON responses, and continuously fetch user-generated topics by mimicking the app’s HTTP requests, complete with sample code and request headers.
This tutorial shows how to use Python's itchat library to extract your WeChat contacts' province and city information, analyze the distribution, and create visual maps, providing step‑by‑step guidance and sample screenshots for each stage.
This tutorial shows how to employ Python's itchat library to log into WeChat, retrieve the total number of friends, and analyze their gender distribution, providing step‑by‑step code, screenshots of results, and tips for verifying the data.
This tutorial walks you through extracting WeChat Moments with a Python web scraper, processing the Chinese text using jieba, and visualizing the most frequent words as beautiful word clouds with customizable shapes and fonts.
Learn how to export WeChat Moments using a third‑party service, set up a Scrapy project in Python, analyze the dynamic JSON responses, and write a crawler to extract timeline data, complete with screenshots and command‑line instructions for a fully functional scraper.
This tutorial demonstrates how to collect Python job postings from Lagou using Python's requests library, process the JSON response with pandas, and create insightful visualizations—including bar charts, word clouds, and geographic heatmaps—while handling anti‑scraping measures and data cleaning steps.
This article introduces Puppeteer, a Node library for controlling Chrome/Chromium, explains its installation, key APIs, and demonstrates practical use cases such as generating PDFs, automating UI tests on mobile emulators, and capturing performance traces for web pages.
A comprehensive catalog of Python libraries covering network communication, web crawling frameworks, HTML/XML parsing, text manipulation, file format handling, natural language processing, browser automation, concurrency, cloud services, email processing, URL manipulation, multimedia extraction, WebSocket support, DNS resolution, computer vision, proxy servers, and other useful tools for developers.
This step‑by‑step tutorial shows Python beginners how to create a simple web scraper that downloads, parses, and stores the Maoyan movie TOP 100 list using requests, regular expressions, and multiprocessing for fast data collection.
This article explains why Python is a top choice for beginners and professionals, outlines its rich library ecosystem, and presents a four‑stage learning path—from basic syntax and data structures to advanced features and real‑world projects—helping readers become competent Python developers.
This article demonstrates a compact, runnable Python 3 scraper that fetches job listings from Lagou.com based on a keyword, filters by city and salary, extracts detailed job requirements using XPath, and applies Baidu's free NLP service for word segmentation and part‑of‑speech tagging to reveal key skill terms.
This guide walks you through logging into Baidu Cloud, capturing the transfer API via browser dev tools, analyzing required cookies and parameters, constructing the proper request headers and URL, and programmatically extracting share information to automate file transfers using Python.
This guide explains how to collect images from Zhihu’s “美女” topic using Python’s Requests and lxml, filter them with Baidu’s AipFace API based on gender, face presence, authenticity, and beauty score, and store the high‑quality results locally, including setup and optional customizations.
This tutorial explains how to build a Python program that crawls 71,000 Tang poems, extracts and tokenizes the text, analyzes word frequencies, and assembles new five‑character regulated verses with proper rhymes, including acrostic poems, while offering code snippets and future AI enhancements.
This article walks through using Python to log into CSDN, capture the dynamic Geek News list via its JSON API, handle request parameters, and extract article titles and links, providing concise code screenshots for a complete web‑scraping solution.
Learn how to use Python's BeautifulSoup to scrape real‑time weather data from Sohu Weather, format the information, and automatically send it via email using SMTP, with step‑by‑step code examples and tips for handling different email providers.
This guide demonstrates how to use Python's BeautifulSoup library to crawl the Green Breath website, extract real‑time weather and PM2.5 information, handle missing data with conditional checks, and display the results directly in the PyCharm console, providing a practical example of web‑scraping for environmental monitoring.
This tutorial shows how to use Python to crawl all Chinese city names from Zhaopin, retrieve the number of Android job postings for each city via HTTP GET requests, parse the results with regex, store them in a dictionary, and finally plot the data with Matplotlib for clear visual comparison.
This article compares four Python web‑scraping selectors—regular expressions, BeautifulSoup, lxml/XPath, and CSS—detailing their strengths, weaknesses, performance, and installation difficulty to help developers choose the most suitable tool for extracting data from sites like JD.com.
This tutorial shows how to use Python's urllib and XPath to accurately extract product details such as name, link, image, and price from JD.com search results, offering a clearer alternative to regular expressions and BeautifulSoup.
This tutorial shows how to use Python's urllib and BeautifulSoup libraries to encode search keywords, request JD.com pages, parse the HTML tree, and reliably extract product names, links, images, and prices, offering a simpler alternative to complex regular‑expression scrapers.
This article explains how to design and build a lightweight Java web crawler framework, covering crawler fundamentals, anti‑scraping challenges, core components such as URL manager, scheduler, downloader, parser and pipeline, and provides concrete code examples and architectural diagrams.
This comprehensive guide walks you through Python web‑scraping techniques—including basic URL fetching, proxy usage, cookie and form handling, browser impersonation, gzip/deflate support, captcha processing, multithreading with thread pools and Twisted async I/O, plus practical tips on connection pooling, thread stack size, retries, timeouts and login automation—providing a solid foundation for building robust crawlers.
This guide explains how to use XPath efficiently for web scraping, covering node selection, axes, functions, numeric comparisons, and advanced combinations, while emphasizing concise and readable expressions to improve performance and maintainability.
This guide demonstrates how to combine Python's regex, BeautifulSoup, and Selenium (including Chrome and headless PhantomJS) for powerful web scraping, covering tag matching, handling Ajax, iFrames, cookie management, and practical code examples for extracting and interacting with dynamic web content.
This article scrapes over 22,000 Douban comments for the TV series “The Hunt,” cleans the data, applies SnowNLP‑based sentiment classification and word‑cloud visualization, and shows that while overall ratings are mixed, detailed short reviews are overwhelmingly positive.
This tutorial walks through using Python's urllib and regular expressions to crawl a Baidu Tieba page, extract all .jpg image URLs, and download each image locally with a sequential naming scheme.
This tutorial explains how a Python developer can set up a Scrapy project, write spiders to crawl Zhihu user profiles, store the results in a MySQL database, adjust settings for headers and delays, and finally perform simple gender and location analysis on the collected data.
This guide walks through using Python's urllib2 library for web crawling, covering basic GET/POST requests, handling proxy IPs, managing cookies, spoofing browser headers, processing gzip-compressed responses, and implementing multithreaded fetching with a simple thread‑pool template.
etlpy is a lightweight Python framework that lets you define web‑crawling and data‑cleaning pipelines via XML, using generators for streaming, built‑in thread pools for parallelism, and a plug‑in architecture that handles everything from regex parsing to JSON conversion, all within a single 500‑line core file.
This tutorial walks through using Python and BeautifulSoup to crawl user profiles, fan lists, and song information from the Quanmin K‑Song app, clean and store the data in MongoDB, handle pagination, and prepare the dataset for further analysis.
This tutorial walks through using Python and browser developer tools to locate, decode, and retrieve the hot comments from NetEase Cloud Music's hot song chart, including extracting song IDs, handling encrypted request parameters, and applying regex to gather song metadata.
This guide walks you through using Python to crawl Qunar’s tourist spot pages, retrieve location and sales data, enrich it with Baidu Map’s geocoding API, and then create interactive heatmaps and ranking charts with ECharts, while storing results in JSON and Excel files for further analysis.
Learn how to set up a Scrapy project to crawl comprehensive car brand, series, and model data from Autohome, covering environment preparation, project initialization, spider and pipeline creation, CSV output, rate limiting, and useful debugging tips.
This article introduces Electron and the Nightmare framework, explains how to install and use Nightmare for web automation and crawling, and provides a complete example of scraping Zhihu topic data with JavaScript, Node.js, and Cheerio, including code snippets and JSON output.
This tutorial walks through building a Python web scraper that automatically downloads images from Baidu by analyzing requirements, inspecting page source, crafting regex patterns, and implementing the crawler with requests, offering step‑by‑step guidance, code snippets, and troubleshooting tips.
Inspired by a tutorial, I created a MongoDB‑backed crawler for the Yisou novel website, extracting category links, managing URL states across multiple processes, handling millions of pages, and finally deduplicating the results to obtain a clean collection of books.
This guide walks through building a Python web crawler to extract novel titles and URLs from the qu.la ranking page, explains the site’s clear HTML structure, shows how to deduplicate entries with a set, and provides complete code snippets plus performance tips and a Scrapy upgrade path.
This article walks through creating a Python-based Zhihu web crawler, detailing the tech stack, data collection, visualization of user demographics and top contributors, the crawler architecture, authorization handling, and suggestions for performance and storage improvements.
This article details a step‑by‑step process for crawling all popular Ximalaya channels, extracting each audio's metadata, and storing roughly 700,000 records in MongoDB, while also showing how to speed up the crawl with asynchronous requests.
This guide walks through the complete process of crawling the Sichuan University Public Administration College website to extract every news article, covering target identification, rule definition, code implementation, handling pagination, and troubleshooting missing items.
This article introduces five highly practical Python packages—yagmail, requests, psutil, BeautifulSoup, and a collection of utility scripts—explaining how each simplifies common tasks such as sending emails, making HTTP calls, system monitoring, web scraping, and code reuse, complete with concise code examples.
This guide explains why websites employ anti‑scraping defenses, outlines the most common header checks such as User‑Agent, Referer, and Cookies, and provides practical Scrapy code snippets for rotating user agents, managing proxies, handling X‑Forwarded‑For, limiting request rates, and dealing with dynamic AJAX content using Selenium or PhantomJS.
This tutorial demonstrates how to capture hot comments from NetEase Cloud Music using web‑scraping techniques, handle the platform's encrypted API, and generate a Chinese word cloud with Python's WordCloud library for visual insight.
This tutorial walks through creating a Chrome extension that automates page clicks to repeatedly refresh and purchase a Huawei Honor V9, covering element inspection, simulated clicks with timers, JavaScript fundamentals, packaging the extension, and additional handy scripts.
This article explains how to use Python's urllib2 module to fetch web pages, covering basic request handling, creating Request objects, sending GET and POST data, setting custom headers, and demonstrates practical examples with code snippets and screenshots.
This guide shows how to collect every @mentioned user on a Zhihu question page by using a JavaScript bookmarklet or a Python script, explains the extraction process, provides the necessary code snippets, and discusses why following programmers on Zhihu may not be the most effective learning method.
This article walks through using PHP's cURL extension to crawl tens of thousands of Zhihu user profiles, parse the HTML with regular expressions, store the extracted data efficiently, and present the results with responsive charts and dashboards.
Web crawlers, especially AI‑driven ones, threaten site performance and data ownership, so this article reviews common anti‑scraping methods—from IP and header analysis to behavior detection—and reveals two unconventional defenses: data poisoning and a deposit‑based access model that penalize malicious bots.
This guide outlines the main anti‑scraping techniques used by websites—including header validation, user‑behavior monitoring, and dynamic content loading—and provides practical methods such as header spoofing, IP proxy rotation, request throttling, and Selenium/PhantomJS automation to overcome them.
This article presents a 16‑week Python data‑analysis learning roadmap covering environment setup, basic syntax, web‑scraping techniques, data‑analysis libraries such as pandas and NumPy, and data‑visualization with matplotlib, along with curated free resources and tutorials for each stage.
This article summarizes common anti‑scraping techniques—including header checks, user‑behavior detection, and dynamic page defenses—and provides practical ways to circumvent them using custom headers, IP proxies, request timing, and tools like Selenium with PhantomJS to simulate real browsers.
This tutorial walks you through setting up a macOS environment, inspecting a web page, and writing a Python script with the requests library to locate and download all images from a target site, complete with code explanations and execution tips.
Discover a comprehensive collection of Python libraries covering network requests, web crawling frameworks, HTML/XML parsing, text manipulation, file format handling, natural language processing, browser automation, asynchronous programming, and more, providing developers with essential tools for efficient web scraping and data processing tasks.
This tutorial walks through collecting over a million Zhihu user profiles using PHP on Ubuntu, handling cookies, bypassing image hot‑link protection, scaling requests with curl_multi, de‑duplicating MySQL inserts, and coordinating work with Redis and multi‑process pcntl for efficient large‑scale web scraping.
This article presents a PHP‑based web‑scraping experiment that collects and visualizes several months of data from the “Da Gai Er” board of the Caoliu forum, revealing user activity patterns, image hosting distribution, registration trends, and overall forum health through charts and statistical summaries.
This article compiles the top Quora answers showcasing Python scripts that automate Facebook birthday thank‑you comments, one‑click subtitle downloads, IMDb data extraction, comic and e‑card scraping, and explains how these projects even helped the author land a job.
This guide walks through practical Python web‑scraping techniques using urllib2, covering basic page fetching, proxy usage, cookie handling for logins, form submission, header spoofing, anti‑hotlink tricks, multithreaded crawling, and strategies for bypassing simple captchas, all illustrated with code snippets.
