This guide walks you through setting up a Python environment, installing required libraries, constructing a FilmSky scraper class, handling pagination, parsing HTML with regex, and saving movie titles and download links, enabling you to browse and download movies from the FilmSky website efficiently.
This article explains how crawler engineers can fetch real‑time data such as sports scores, stock quotes, or cryptocurrency prices by comparing polling and WebSocket approaches, introducing the aiowebsocket library, and providing complete Python code to perform handshake, subscription, and continuous data streaming.
This tutorial shows how to locate the "next page" link on a website, extract its URL using Scrapy selectors, add proper checks, and integrate the pagination logic into a Scrapy spider so that all article pages are crawled automatically.
This tutorial explains how to use Scrapy's Request objects to feed article detail URLs into the crawler, configure callbacks for parsing, handle relative URLs with urljoin, and yield requests so Scrapy can download pages, completing the core data extraction workflow.
This article demonstrates how to scrape the Hurun Rich List from 2015 to 2020 using Python, clean the data, and create interactive visualizations of the top 20 wealth holders, wealth trends over six years, and industry shifts with Pyecharts.
This guide walks you through installing Scrapy-Distributed, setting up RabbitMQ and RedisBloom containers, creating a sitemap spider, configuring the distributed scheduler and dupefilter, and running the spider, while explaining why this non‑intrusive solution improves over existing Scrapy‑Redis and scrapy‑rabbitmq approaches.
This article continues a Scrapy tutorial by showing how to extract like counts, comment counts, and full article content using XPath selectors, regular expressions, and debugging techniques, providing step‑by‑step code examples and screenshots to help Python developers automate web data collection.
This tutorial shows how to apply XPath expressions within the Scrapy framework to extract titles, publication dates, tags, content, likes, favorites, and comments from a sample website, providing practical code snippets and tips for reliable web data collection.
This tutorial explains how to use Python to scrape Wallhaven for 4K wallpapers, automatically iterate through pages, download images in bulk, and handle request headers and delays to avoid blocking, providing complete code and result screenshots.
Learn how to efficiently build Python web scrapers by mastering basic GET and POST requests, configuring proxy IPs, handling cookies, spoofing browser headers, enabling gzip compression, and leveraging multithreaded concurrency to accelerate data extraction.
This tutorial walks through using Selenium to handle pop‑ups and simple numeric captchas on a web portal, captures the captcha image, applies binary thresholding, recognizes the text with Tesseract OCR, and then submits the login credentials, including retry logic for failed recognitions.
This tutorial walks you through using Python's requests, lxml, and JSON parsing to automatically crawl the King of Glory website, extract hero skin URLs, and download the images into organized folders while displaying progress in the console.
This article walks through creating a Python web scraper that extracts financial data for over three thousand listed companies, starting from a simple pandas script and progressively adding error handling, MySQL storage, and multiprocessing to build a robust, production‑ready tool.
Using Python Selenium, the author collected 4,403 Taobao listings for men's shorts, cleaned the data, visualized regional sales, price distribution, top shops, and product characteristics, ultimately identifying the best-selling items and revealing market trends.
This tutorial shows how to use Python's urllib and json libraries to call the Kuaidi100 API, retrieve real‑time logistics data for various Chinese courier companies, and display the tracking timeline, while explaining how to discover the correct request URL via browser dev tools.
This article demonstrates how to use Python to crawl 3,032 Chinese recipe entries from Douguo.com, clean the data with Pandas, and create insightful visualizations—including rating distributions, cuisine comparisons, and ingredient word clouds—using pyecharts, providing complete code snippets and analysis of the results.
This article demonstrates how to use Python to crawl over 200,000 Bilibili comments, analyze popular memes across different content categories, and presents a step‑by‑step guide to extracting danmu via network requests with code examples for practical data mining.
This tutorial introduces Python's Requests library, covering installation, basic GET and POST requests, adding headers and parameters, handling proxies, cookies, sessions, and disabling SSL verification, with clear code examples for each feature.
This article explains why Python developers often need to run JavaScript—especially obfuscated code—during web scraping, demonstrates how to use the ExecJS library to execute both simple and heavily obfuscated JavaScript from Python, and shows that the approach works seamlessly without de‑obfuscation.
The article demonstrates how to use Puppeteer—a Node.js API built on the Chrome DevTools Protocol—to run headless Chrome for tasks such as scraping Google emoji images, generating screenshots or PDFs, and automating front‑end tests by launching a browser, navigating pages, handling cookies, simulating user input, capturing responses, and saving results.
This tutorial explains four practical methods—PyExecJS, js2py, Node.js, and PyV8—to execute JavaScript code from Python, providing code examples and tips for handling encrypted parameters during web crawling.
This tutorial walks you through creating a Python web scraper that extracts images from a target site, handles anti‑scraping measures, saves them into categorized folders, and logs the download results, while explaining the required libraries, code structure, and best practices.
This article reviews eight essential Python tools—including IDLE, Scikit‑learn, Theano, Selenium, TestComplete, BeautifulSoup, Pandas, and PuLP—explaining their main features, typical use cases, and why they are valuable for developers and students across web, data science, automation, and optimization tasks.
Learn how to build a Python web scraper that fetches upcoming movie details from Maoyan.com, covering environment setup, URL pagination, random User‑Agent handling, HTML parsing with XPath, data extraction, and result display, while highlighting best practices and limitations.
This tutorial explains how to analyze Meituan food page URLs, use browser developer tools to locate AJAX JSON responses, construct Python requests with proper headers, extract restaurant information via regular expressions, and save the results to a local file.
This article analyzes 13,966 Chinese operations‑engineer job postings scraped from 51job, cleaning the data with Python and Pandas, then visualizing industry demand, city concentration, salary ranges, education requirements, company size distribution, and keyword trends to guide job seekers and recruiters.
This article analyzes 13,966 Chinese operations engineering job postings collected from 51job, detailing scraping methods, data cleaning steps, and visualizations that uncover top hiring industries, city demand, salary ranges, required education, company size distribution, and keyword trends for the ops market.
This tutorial explains how to crawl 58.com job listings with Python, extract location, company, and salary information, handle anti‑scraping defenses using realistic headers and random User‑Agents, and save the results into a text file.
This tutorial introduces Python’s Requests library, covering installation, core methods like GET, POST, PUT, PATCH, DELETE, detailed parameters, session handling, exception management, header customization, proxy usage, and practical code examples to empower effective web scraping.
The article demonstrates how to scrape over 200,000 iQiyi danmu comments for the drama “The Bad Kids” using Python, then analyzes user activity, episode popularity, top comments, actor mentions, and visualizes the results with word clouds and charts.
This article demonstrates how to scrape bullet comments from iQiyi for the first episode of a popular mystery series, decode the binary files, extract the text, and use Python's jieba and wordcloud libraries to clean the data and generate a visual word cloud of audience sentiments.
This article demonstrates how to scrape BI engineer job listings from Zhaopin.com using Python, clean and enrich the data, and create visual analytics with FineBI to reveal salary trends, city distribution, experience levels, and education requirements in the Chinese job market.
Using Python to scrape 4,277 Zhihu answers about changing WeChat IDs, this article shows how the data was collected, presents the extraction code, and uncovers six surprising patterns in users' chosen usernames, illustrating social trends behind the recent feature update.
The article provides a comprehensive guide to Python libraries essential for data analysis, detailing tools for data acquisition (Selenium, Scrapy, Beautiful Soup), cleaning (spaCy, NumPy, pandas), visualization (Matplotlib, Pyecharts), modeling (scikit‑learn, PyTorch, TensorFlow), model inspection (LIME), audio (Librosa), image processing (OpenCV, scikit‑image), database access (PyMongo) and web deployment (Flask, Django).
This tutorial walks through creating a Python script that scrapes video URLs from a web page, downloads them in batches with proper buffering, and displays a real‑time progress bar, covering page analysis, code implementation, and practical tips.
This tutorial walks you through building a Python web scraper that extracts recipe names, ingredients, and download links from the XiaChuFang cooking website, handling anti‑scraping measures with custom headers and fake user agents, and saves the collected data into a Word document for future use.
This article explains the concept and benefits of CSS sprites, analyzes their drawbacks for web performance and security, and provides a step‑by‑step Python‑based method—including code snippets—to extract and sum numbers hidden behind sprite images used as an anti‑scraping measure.
This tutorial walks you through creating a Python web scraper that fetches image URLs from 51miz.com using requests and lxml, filters them with regular expressions, downloads the images, and demonstrates the complete workflow with code snippets and screenshots.
This guide walks through building a Python web‑scraper that fetches house names, prices and popularity from Lianjia, formats the data into a Word template, and generates individual Word documents while demonstrating key libraries, URL handling, HTML parsing, and best‑practice considerations.
This article describes how to discover Bilibili's comment XML APIs, decode the embedded CRC32‑hashed user IDs, choose an appropriate database schema, and implement a Python‑PHP tool that retrieves, filters, and displays comments based on video CID and keyword.
This tutorial demonstrates how to use Python's requests, threading, and queue modules to build a multithreaded crawler that extracts game names, download links, and execution time from the Xiaomi App Store, complete with code examples and performance tips.
This article introduces Python's re module, explains core regex functions such as match, search, replace, and compile, demonstrates practical code examples with images, and shows how to apply regular expressions for web‑scraping tasks.
This guide explains several techniques—including using browser developer tools, console commands, and a Windows utility—to copy protected text from websites and download documents like Baidu Docs, while noting their limitations and required steps.
This tutorial explains how to crawl all League of Legends hero skin images with a Python script, decode the URL pattern, download the assets, and then assemble them into a large photo mosaic using a third‑party mosaic software, providing full code and step‑by‑step instructions.
This tutorial explains how to build a Python web‑scraper that searches Baidu Tieba by keyword, bypasses anti‑crawling measures, extracts image and video URLs with XPath, and saves the media files locally, complete with code examples and setup instructions.
This article examines Douban ratings and 500 user comments for the Chinese web drama "Dragon Tomb Cave", visualizes rating distribution, comment timing, geographic origins, and compares sentiment with other series using Python web‑scraping and data‑visualization techniques.
This tutorial shows how to use the Python‑based you‑get tool to download videos, images, and music from dozens of popular sites with a single command, adjust format and quality, and install the utility via pip, Git, or Homebrew.
This guide walks you through setting up a Python environment, installing required libraries, writing a FilmSky class with request handling, parsing HTML using regular expressions, and saving movie titles and download links, providing a practical example of web crawling for movie sites.
This tutorial walks you through extracting image URLs from a webpage using XPath, constructing full URLs, and automating batch downloads with Python's requests, lxml, and fake_useragent libraries, including code snippets and practical tips for handling files and headers.
This article provides a comprehensive step‑by‑step guide to installing Scrapy, understanding its core components and processing flow, creating a weather‑data crawling project, writing items, settings, middlewares, spiders, running the crawler, exporting results, and storing the scraped data into MongoDB.
This article explores the probability of successful bottom‑buying in the stock market by building a data model, using Python to scrape fund data, analyzing individual and multiple funds, and visualizing results to reveal that only about one in four attempts succeed.
Learn how to install the Web Scraper Chrome extension, configure sitemaps and selectors, and handle pagination, scrolling, and click‑load scenarios to build robust, multi‑level web crawling projects without writing code, covering selector types, multi‑page navigation, and data export to CSV.
This article demonstrates how to use Go with the chromedp library to drive headless Chrome for element screenshots, full‑page captures, PDF generation, and device emulation, offering a lightweight alternative to Selenium‑based solutions.
This guide explains four practical methods—using a known cookie, simulating login with urllib or requests, maintaining a session, and employing a headless Selenium browser—to programmatically retrieve pages that require user authentication, complete with step‑by‑step instructions and code examples.
This article details how to use Python to crawl user and article data from the Jianshu platform, then apply FineBI for business intelligence analysis and visualizations, covering author contracts, follower distribution, popular articles, and engagement metrics.
This article explains why personalized tech‑news gathering is valuable, proposes a DIY framework for controlling sources, collection, filtering, reading experience and iteration, and demonstrates three concrete Node.js scraping examples—HTML pages, API data, and WeChat public accounts—plus extended thoughts on building a simple product.
This tutorial explains how to obtain free proxy IPs, extract their addresses using Python's requests and BeautifulSoup, and continuously validate them to overcome anti‑scraping restrictions when crawling sites such as Baidu Baike for data mining tasks.
This tutorial introduces a lightweight Python web crawler that, without relying on third‑party scraping frameworks, fetches high‑resolution skin images and all hero background stories from a popular game, explaining required libraries, implementation steps, and showing the resulting outputs.
This article provides a comprehensive introduction to Python web scraping, covering the fundamental concepts of spiders, HTTP/HTTPS protocols, the Requests library usage, custom headers, proxies, cookies, and various data extraction techniques such as JSON parsing, XPath, and regular expressions.
This article reviews eight PHP web‑scraping libraries—Goutte, Simple HTML DOM, htmlSQL, cURL, Request, HTTPful, Buzz, and Guzzle—detailing their features, requirements, licensing, and documentation to help developers choose the right tool for their backend data‑extraction projects.
This guide introduces Scrapy, a powerful Python web‑scraping framework, explains its architecture and components, walks through installation, project creation, spider development, query syntax, recursive crawling, and item pipelines, providing practical code examples for building robust crawlers.
This article explains how to build a self‑healing dynamic proxy pool for 24/7 web crawling using Scrapy and Redis, covering requirements, design, implementation details, deployment steps, and a reusable Scrapy middleware example.
This guide explains practical techniques to avoid IP bans and 403 errors when web‑scraping, covering explicit and implicit waiting, User‑Agent spoofing, proxy usage, IP pools, and distributed crawling architectures.
When the Chinese railway ticketing system 12306 crashes under heavy load, developers turn to open‑source Python bots that simulate user behavior, query seat availability, and automate order submission, with detailed feature lists, repository links, and real‑world log examples.
This tutorial walks you through planning, extracting, and saving classic Chinese novel content from shicimingju.com using Python, regular expressions, and file storage, providing clear code examples and practical tips for successful web scraping.
This guide introduces Python’s Requests library, covering installation, GET and POST requests, handling headers, response codes, cookies, redirects, timeouts, and proxy settings, with practical code examples to help developers perform reliable HTTP operations and avoid common pitfalls in web scraping.
This article explains the legal boundaries of web crawling in China, covering recent cybersecurity regulations, what makes a crawler illegal or legal, common developer questions, and practical advice to avoid personal‑data violations and criminal liability.
This article explains how to scrape Chinese Basketball Association (CBA) match data from a portal website, analyzes the page structure, extracts table rows using regular expressions, converts them to CSV format, and provides a complete Java/Groovy code example for automated data collection.
This article presents seven practical Python scripts—from scraping Zhihu images and chatting bots to poetry author detection, lottery generation, auto‑drafting apologies, screen recording, and GIF creation—showcasing how to quickly automate diverse tasks without reinventing the wheel.
This article explains how to use a Groovy script to crawl over 7 million historical weather entries for 3,200 cities spanning 2011‑2019, process the JSON responses, and store the cleaned data into a MySQL table, while sharing practical tips and code snippets.
This guide explains practical techniques such as faking User‑Agent headers, rotating cookies, adding random delays, and using proxy pools to prevent IP bans while crawling large amounts of data from websites with anti‑scraping defenses.
This guide walks through using Python's requests and BeautifulSoup libraries to simulate Zhihu login, extract user profiles, retrieve answer likers, download avatars, fetch all answers for a question, and store the collected data in a SQLite database.
This tutorial walks you through using Python to collect tourism data from Qunar, extract key fields such as name, price, and rating, store the results in Excel with pandas, and visualize sales and popularity trends using pyecharts, including a simple recommendation algorithm.
This article introduces Puppeteer, a Node library for controlling Chrome, and demonstrates its use in automating web performance analysis and scraping tasks.
This article demonstrates how to use Selenium and Python to crawl over 6,000 Chinese tourist attractions from Qunar, extract ratings, popularity and sales data, and visualize the results with pandas, seaborn, matplotlib, and pyecharts, revealing the most visited sites and regional travel trends during the 2019 National Day holiday.
This article walks through the challenges of scraping Dazhong Dianping, explains how the site hides numeric data with custom CSS fonts, and provides a complete Python workflow—including HTTP requests, font extraction, glyph rendering, and OCR—to decode and retrieve the protected information.
This guide explains core XPath concepts—including node selection shortcuts, predicate filters, wildcard usage, multiple path unions, axis navigation, and built‑in functions—providing clear examples for effective XML/HTML data extraction.
After uncovering an unprotected API that allowed unlimited resource access, the author created a rough Java program that uses a fixed-size thread pool and CountDownLatch to fetch 100 000 items in parallel, retrieving 10 000 records per thread via HTTP GET requests.
The article walks through a Java solution for extracting multilingual data from an HTML table using regular expressions, handling spacing and encoding issues, splitting fields, and constructing INSERT statements to populate a country_code database table.
This tutorial walks you through logging into Taobao with Python requests, handling anti‑scraping measures, extracting product information via the PC search API, parsing JSON data, and saving the results to Excel, while also covering common pitfalls like sliders and proxy management.
Learn a step‑by‑step Python solution to locate, download, and store high‑resolution, royalty‑free images from ColorHub by navigating its three‑tier page structure, generating request headers, parsing HTML with BeautifulSoup, and saving files locally, enabling offline PPT creation without copyright concerns.
This tutorial introduces BeautifulSoup, a powerful Python library for parsing HTML and XML, covering installation, basic usage, tag selection, attribute extraction, navigation of parent and sibling nodes, method and CSS selectors, and best‑practice recommendations for efficient web data extraction.
This tutorial walks you through creating a Python‑based GitHub monitoring tool that logs in, crawls code search results for sensitive keywords, extracts repository details, writes findings to CSV, and sends email alerts, providing a practical approach to detecting accidental source‑code leaks.
The article explains how to build a poster‑rendering service with Puppeteer, detailing its advantages over canvas, the Redis‑based caching and CDN workflow, optimization tricks for headless Chromium, and future plans to boost QPS and pre‑generate popular posters.
This article walks through a complete data‑analysis workflow—scraping Zhihu dating‑preference answers, cleaning and filtering the data, deriving gender and activity metrics, designing a four‑step screening process, and finally ranking candidates with a custom like‑to‑comment index—to help a single programmer create a concise, high‑quality list of potential partners.
This article walks through collecting Python job postings from Lagou by handling anti‑scraping measures, parsing POST requests, storing results in Excel, and then using pyecharts to create bar, map, and pie visualizations that reveal city distribution, salary ranges, and experience requirements.
The article explains how to create a WeChat Mini Program backend with Tencent Cloud development, use the React‑based Taro framework to build a Japanese sentence helper, consolidate multiple cloud functions via tcb-router, and scrape example sentences with superagent and cheerio, highlighting setup tips and known limitations.
This tutorial explains how to extract Meituan Takeout app comments by analyzing Ajax requests, constructing dynamic URLs, parsing JSON with regular expressions, and storing the results in text files and MongoDB using Python.
This guide explains why Scrapy on Windows raises the "No module named win32api" error, walks through installing the correct pywin32 package (or pypiwin32), shows how to obtain the proper wheel from an unofficial source, and provides extra tips for locating Scrapy spider names.
This article details a complete workflow for extracting nationwide tourist attraction data from Qunar, cleaning and enriching it with geographic coordinates, and performing multi‑level statistical analysis and visualizations—including sales rankings, popularity metrics, heatmaps, and word clouds—to reveal regional tourism patterns across China.
This tutorial demonstrates how to use asynchronous Python (aiohttp, motor, asyncio) to scrape all comments from a popular Bilibili video, store them in MongoDB, and generate a Chinese word‑cloud analysis to reveal why the video went viral.
This article walks through building a Python Scrapy spider to extract comprehensive car brand, series, and model data from Autohome, covering environment setup, project initialization, spider and item definitions, handling lazy-loaded pages, CSV output configuration, rate limiting, user‑agent rotation, and debugging tips.
This guide explains the core XPath syntax—including node selectors, predicates, wildcards, multiple paths, axes, and built‑in functions—providing clear examples so you can efficiently locate and filter XML/HTML elements when scraping web pages.
Using Python’s Scrapy, pandas, and seaborn, the author scraped and cleaned public Shanbay user data, stored it in PostgreSQL, and analyzed registration and study habits to reveal that over 68% of users abandon word‑learning on day one, with only a tiny fraction persisting beyond 100 days.
This article shows how to use Python to scrape Zhihu answers, store them in MongoDB, filter for short high‑upvote replies, and then presents a collection of programmer‑centric jokes that illustrate the kind of "god replies" the crawler can retrieve.
This guide explains how to extract novel titles and links from a structured ranking website, remove duplicate entries using a set, handle HTML tags, and improve crawling speed with multithreading or the Scrapy framework, all while keeping the code modular and reusable.
This guide explains how to scale web crawling to hundreds of sites using Scrapy‑Redis, covering its components, distributed workflow, Redis installation and configuration, proxy pool handling, and provides complete Python code examples for spiders and pipelines.
This article explores headless browser automation technologies including Selenium, PhantomJS, Puppeteer, and Headless Chrome, comparing their architectures, use cases, and implementation differences.
This article describes how to use Python, Selenium, and search‑index services to scrape and visualize popularity data for Chinese tourist spots during the National Day holiday, presenting a ranking of destinations and providing full code examples for data collection, cleaning, and storage.
This article explains why regular expressions are essential for Python web scraping, introduces the special characters ^, ., and *, and demonstrates their use with clear code examples, showing how to extract specific patterns such as numbers from HTML content.
