This article provides a comprehensive guide to Python web scraping, covering basic GET/POST requests, proxy usage, cookie management, browser header spoofing, gzip compression handling, and multithreaded crawling to improve efficiency and avoid common obstacles.
This guide walks you through configuring PyCharm to connect to a remote Linux server, set up an SSH Python interpreter, enable automatic file synchronization, and run long‑running web‑scraping scripts directly on the server, streamlining development without manual file transfers.
Learn step‑by‑step how to scrape Tencent’s location data via its API, process the coordinates with pandas, deduplicate and filter them, and finally visualize regional foot‑traffic as an interactive heatmap using Folium, complete with code snippets and practical tips.
Using Python, this tutorial demonstrates how to scrape 1,585 cherry product listings from Taobao, extract key fields, clean and transform the data, and visualize regional sales and price distributions, highlighting the most popular provinces, stores, and price ranges, all with reproducible code.
This article introduces Microsoft’s open‑source Playwright‑Python library, explains its advantages over Selenium, walks through installation, code generation, synchronous and asynchronous usage, mobile emulation, and highlights its strengths and current limitations for Python developers.
In this article we demonstrate how to use Python’s requests and BeautifulSoup to scrape a static webpage listing the world’s top 100 most beautiful women, process the data with pandas and pyecharts for continent, nationality and profession statistics, and apply Baidu’s face‑recognition API to score each celebrity’s attractiveness.
This tutorial walks you through installing Scrapy on Windows, creating a project and spider, configuring settings, and using XPath to crawl and extract joke titles and contents from the Qiushibaike website, providing a solid foundation for Python web scraping.
This guide demonstrates how to retrieve a list of WeChat public account articles by constructing the appropriate API URL, examining its query parameters, viewing the JSON response, and using a Python script to parse the data and generate a markdown file of titles and links.
This article demonstrates how to use Python's requests, fake_useragent, and lxml libraries to batch‑scrape nearly a thousand new‑house listings from the 惠民之家 website, extracting 41 fields such as name, price, layout, opening date, plot ratio and green ratio, while handling pagination and anti‑scraping measures.
This tutorial explains how to use Python's requests, lxml, and urllib libraries to search Baidu Tieba by keyword, bypass anti‑crawling measures, extract image and video URLs with XPath, and save the media files locally, complete with code examples and screenshots.
This article introduces the fundamentals of XPath syntax, common rules, and example expressions, then explains how to use the lxml library in Python for HTML/XML parsing, including practical tips and a complete code example for extracting links and text from a sample document.
This tutorial shows how to use Python to fetch data from Bilibili’s “马保国” channel via its public API, extract video metadata, clean and visualize 14,000 records, and generate insights such as top‑viewed videos and a comment word cloud.
Learn how to build a Python web scraper that extracts iQIYI movie titles, actors, and scores, parses the data with regex or BeautifulSoup, displays results in a Tkinter GUI with a combobox, and saves the information to a file—all explained with code snippets and screenshots.
This article demonstrates how to crawl 6,025 Beijing Dank Apartment listings using Python, clean and enrich the data with Pandas, and visualize distribution, price, size, floor, and subway proximity through charts, revealing key market insights and correlation patterns.
This article explains how to locate Weibo comment APIs, work around rate limits by using the mobile site, extract required parameters, and implement a Python script that handles cookies, pagination, emoji cleaning, deduplication, and scheduled execution to collect comments efficiently.
This article provides a step‑by‑step guide on using Scrapy’s crawl template to automatically scrape product details such as ID, title, shop name, shop link, and price from JD.com, including source analysis, project setup, code snippets, and result verification.
This article walks through the complete process of designing a Python web scraper to collect Zhihu answer data, parse author, ID and excerpt fields, store the results in CSV files, and finally visualize the text with a word‑cloud, including all necessary code snippets and explanations.
This article walks through using Python to crawl Tencent Video's "Offer" season 2 comments, merge and clean the CSV data, perform exploratory analysis, generate visualizations and word clouds, and apply Baidu's open‑source NLP model for sentiment scoring, providing complete code snippets for each step.
This guide walks you through setting up a Python environment, installing required libraries, constructing a FilmSky scraper class, handling pagination, parsing HTML with regex, and saving movie titles and download links, enabling you to browse and download movies from the FilmSky website efficiently.
This article explains how crawler engineers can fetch real‑time data such as sports scores, stock quotes, or cryptocurrency prices by comparing polling and WebSocket approaches, introducing the aiowebsocket library, and providing complete Python code to perform handshake, subscription, and continuous data streaming.
This tutorial shows how to locate the "next page" link on a website, extract its URL using Scrapy selectors, add proper checks, and integrate the pagination logic into a Scrapy spider so that all article pages are crawled automatically.
This tutorial explains how to use Scrapy's Request objects to feed article detail URLs into the crawler, configure callbacks for parsing, handle relative URLs with urljoin, and yield requests so Scrapy can download pages, completing the core data extraction workflow.
This article demonstrates how to scrape the Hurun Rich List from 2015 to 2020 using Python, clean the data, and create interactive visualizations of the top 20 wealth holders, wealth trends over six years, and industry shifts with Pyecharts.
This guide walks you through installing Scrapy-Distributed, setting up RabbitMQ and RedisBloom containers, creating a sitemap spider, configuring the distributed scheduler and dupefilter, and running the spider, while explaining why this non‑intrusive solution improves over existing Scrapy‑Redis and scrapy‑rabbitmq approaches.
This article continues a Scrapy tutorial by showing how to extract like counts, comment counts, and full article content using XPath selectors, regular expressions, and debugging techniques, providing step‑by‑step code examples and screenshots to help Python developers automate web data collection.
This tutorial shows how to apply XPath expressions within the Scrapy framework to extract titles, publication dates, tags, content, likes, favorites, and comments from a sample website, providing practical code snippets and tips for reliable web data collection.
This tutorial explains how to use Python to scrape Wallhaven for 4K wallpapers, automatically iterate through pages, download images in bulk, and handle request headers and delays to avoid blocking, providing complete code and result screenshots.
Learn how to efficiently build Python web scrapers by mastering basic GET and POST requests, configuring proxy IPs, handling cookies, spoofing browser headers, enabling gzip compression, and leveraging multithreaded concurrency to accelerate data extraction.
This tutorial walks through using Selenium to handle pop‑ups and simple numeric captchas on a web portal, captures the captcha image, applies binary thresholding, recognizes the text with Tesseract OCR, and then submits the login credentials, including retry logic for failed recognitions.
This tutorial walks you through using Python's requests, lxml, and JSON parsing to automatically crawl the King of Glory website, extract hero skin URLs, and download the images into organized folders while displaying progress in the console.
This article walks through creating a Python web scraper that extracts financial data for over three thousand listed companies, starting from a simple pandas script and progressively adding error handling, MySQL storage, and multiprocessing to build a robust, production‑ready tool.
Using Python Selenium, the author collected 4,403 Taobao listings for men's shorts, cleaned the data, visualized regional sales, price distribution, top shops, and product characteristics, ultimately identifying the best-selling items and revealing market trends.
This tutorial shows how to use Python's urllib and json libraries to call the Kuaidi100 API, retrieve real‑time logistics data for various Chinese courier companies, and display the tracking timeline, while explaining how to discover the correct request URL via browser dev tools.
This article demonstrates how to use Python to crawl 3,032 Chinese recipe entries from Douguo.com, clean the data with Pandas, and create insightful visualizations—including rating distributions, cuisine comparisons, and ingredient word clouds—using pyecharts, providing complete code snippets and analysis of the results.
This article demonstrates how to use Python to crawl over 200,000 Bilibili comments, analyze popular memes across different content categories, and presents a step‑by‑step guide to extracting danmu via network requests with code examples for practical data mining.
This tutorial introduces Python's Requests library, covering installation, basic GET and POST requests, adding headers and parameters, handling proxies, cookies, sessions, and disabling SSL verification, with clear code examples for each feature.
This article explains why Python developers often need to run JavaScript—especially obfuscated code—during web scraping, demonstrates how to use the ExecJS library to execute both simple and heavily obfuscated JavaScript from Python, and shows that the approach works seamlessly without de‑obfuscation.
The article demonstrates how to use Puppeteer—a Node.js API built on the Chrome DevTools Protocol—to run headless Chrome for tasks such as scraping Google emoji images, generating screenshots or PDFs, and automating front‑end tests by launching a browser, navigating pages, handling cookies, simulating user input, capturing responses, and saving results.
This tutorial explains four practical methods—PyExecJS, js2py, Node.js, and PyV8—to execute JavaScript code from Python, providing code examples and tips for handling encrypted parameters during web crawling.
This tutorial walks you through creating a Python web scraper that extracts images from a target site, handles anti‑scraping measures, saves them into categorized folders, and logs the download results, while explaining the required libraries, code structure, and best practices.
This article reviews eight essential Python tools—including IDLE, Scikit‑learn, Theano, Selenium, TestComplete, BeautifulSoup, Pandas, and PuLP—explaining their main features, typical use cases, and why they are valuable for developers and students across web, data science, automation, and optimization tasks.
Learn how to build a Python web scraper that fetches upcoming movie details from Maoyan.com, covering environment setup, URL pagination, random User‑Agent handling, HTML parsing with XPath, data extraction, and result display, while highlighting best practices and limitations.
This tutorial explains how to analyze Meituan food page URLs, use browser developer tools to locate AJAX JSON responses, construct Python requests with proper headers, extract restaurant information via regular expressions, and save the results to a local file.
This article analyzes 13,966 Chinese operations‑engineer job postings scraped from 51job, cleaning the data with Python and Pandas, then visualizing industry demand, city concentration, salary ranges, education requirements, company size distribution, and keyword trends to guide job seekers and recruiters.
This article analyzes 13,966 Chinese operations engineering job postings collected from 51job, detailing scraping methods, data cleaning steps, and visualizations that uncover top hiring industries, city demand, salary ranges, required education, company size distribution, and keyword trends for the ops market.
This tutorial explains how to crawl 58.com job listings with Python, extract location, company, and salary information, handle anti‑scraping defenses using realistic headers and random User‑Agents, and save the results into a text file.
This tutorial introduces Python’s Requests library, covering installation, core methods like GET, POST, PUT, PATCH, DELETE, detailed parameters, session handling, exception management, header customization, proxy usage, and practical code examples to empower effective web scraping.
The article demonstrates how to scrape over 200,000 iQiyi danmu comments for the drama “The Bad Kids” using Python, then analyzes user activity, episode popularity, top comments, actor mentions, and visualizes the results with word clouds and charts.
This article demonstrates how to scrape bullet comments from iQiyi for the first episode of a popular mystery series, decode the binary files, extract the text, and use Python's jieba and wordcloud libraries to clean the data and generate a visual word cloud of audience sentiments.
This article demonstrates how to scrape BI engineer job listings from Zhaopin.com using Python, clean and enrich the data, and create visual analytics with FineBI to reveal salary trends, city distribution, experience levels, and education requirements in the Chinese job market.
Using Python to scrape 4,277 Zhihu answers about changing WeChat IDs, this article shows how the data was collected, presents the extraction code, and uncovers six surprising patterns in users' chosen usernames, illustrating social trends behind the recent feature update.
The article provides a comprehensive guide to Python libraries essential for data analysis, detailing tools for data acquisition (Selenium, Scrapy, Beautiful Soup), cleaning (spaCy, NumPy, pandas), visualization (Matplotlib, Pyecharts), modeling (scikit‑learn, PyTorch, TensorFlow), model inspection (LIME), audio (Librosa), image processing (OpenCV, scikit‑image), database access (PyMongo) and web deployment (Flask, Django).
This tutorial walks through creating a Python script that scrapes video URLs from a web page, downloads them in batches with proper buffering, and displays a real‑time progress bar, covering page analysis, code implementation, and practical tips.
This tutorial walks you through building a Python web scraper that extracts recipe names, ingredients, and download links from the XiaChuFang cooking website, handling anti‑scraping measures with custom headers and fake user agents, and saves the collected data into a Word document for future use.
This article explains the concept and benefits of CSS sprites, analyzes their drawbacks for web performance and security, and provides a step‑by‑step Python‑based method—including code snippets—to extract and sum numbers hidden behind sprite images used as an anti‑scraping measure.
This tutorial walks you through creating a Python web scraper that fetches image URLs from 51miz.com using requests and lxml, filters them with regular expressions, downloads the images, and demonstrates the complete workflow with code snippets and screenshots.
This guide walks through building a Python web‑scraper that fetches house names, prices and popularity from Lianjia, formats the data into a Word template, and generates individual Word documents while demonstrating key libraries, URL handling, HTML parsing, and best‑practice considerations.
This article describes how to discover Bilibili's comment XML APIs, decode the embedded CRC32‑hashed user IDs, choose an appropriate database schema, and implement a Python‑PHP tool that retrieves, filters, and displays comments based on video CID and keyword.
This tutorial demonstrates how to use Python's requests, threading, and queue modules to build a multithreaded crawler that extracts game names, download links, and execution time from the Xiaomi App Store, complete with code examples and performance tips.
This article introduces Python's re module, explains core regex functions such as match, search, replace, and compile, demonstrates practical code examples with images, and shows how to apply regular expressions for web‑scraping tasks.
This guide explains several techniques—including using browser developer tools, console commands, and a Windows utility—to copy protected text from websites and download documents like Baidu Docs, while noting their limitations and required steps.
This tutorial explains how to crawl all League of Legends hero skin images with a Python script, decode the URL pattern, download the assets, and then assemble them into a large photo mosaic using a third‑party mosaic software, providing full code and step‑by‑step instructions.
This tutorial explains how to build a Python web‑scraper that searches Baidu Tieba by keyword, bypasses anti‑crawling measures, extracts image and video URLs with XPath, and saves the media files locally, complete with code examples and setup instructions.
This article examines Douban ratings and 500 user comments for the Chinese web drama "Dragon Tomb Cave", visualizes rating distribution, comment timing, geographic origins, and compares sentiment with other series using Python web‑scraping and data‑visualization techniques.
This tutorial shows how to use the Python‑based you‑get tool to download videos, images, and music from dozens of popular sites with a single command, adjust format and quality, and install the utility via pip, Git, or Homebrew.
This guide walks you through setting up a Python environment, installing required libraries, writing a FilmSky class with request handling, parsing HTML using regular expressions, and saving movie titles and download links, providing a practical example of web crawling for movie sites.
This tutorial walks you through extracting image URLs from a webpage using XPath, constructing full URLs, and automating batch downloads with Python's requests, lxml, and fake_useragent libraries, including code snippets and practical tips for handling files and headers.
This article provides a comprehensive step‑by‑step guide to installing Scrapy, understanding its core components and processing flow, creating a weather‑data crawling project, writing items, settings, middlewares, spiders, running the crawler, exporting results, and storing the scraped data into MongoDB.
This article explores the probability of successful bottom‑buying in the stock market by building a data model, using Python to scrape fund data, analyzing individual and multiple funds, and visualizing results to reveal that only about one in four attempts succeed.
Learn how to install the Web Scraper Chrome extension, configure sitemaps and selectors, and handle pagination, scrolling, and click‑load scenarios to build robust, multi‑level web crawling projects without writing code, covering selector types, multi‑page navigation, and data export to CSV.
This article demonstrates how to use Go with the chromedp library to drive headless Chrome for element screenshots, full‑page captures, PDF generation, and device emulation, offering a lightweight alternative to Selenium‑based solutions.
This guide explains four practical methods—using a known cookie, simulating login with urllib or requests, maintaining a session, and employing a headless Selenium browser—to programmatically retrieve pages that require user authentication, complete with step‑by‑step instructions and code examples.
This article details how to use Python to crawl user and article data from the Jianshu platform, then apply FineBI for business intelligence analysis and visualizations, covering author contracts, follower distribution, popular articles, and engagement metrics.
This article explains why personalized tech‑news gathering is valuable, proposes a DIY framework for controlling sources, collection, filtering, reading experience and iteration, and demonstrates three concrete Node.js scraping examples—HTML pages, API data, and WeChat public accounts—plus extended thoughts on building a simple product.
This tutorial explains how to obtain free proxy IPs, extract their addresses using Python's requests and BeautifulSoup, and continuously validate them to overcome anti‑scraping restrictions when crawling sites such as Baidu Baike for data mining tasks.
This tutorial introduces a lightweight Python web crawler that, without relying on third‑party scraping frameworks, fetches high‑resolution skin images and all hero background stories from a popular game, explaining required libraries, implementation steps, and showing the resulting outputs.
This article provides a comprehensive introduction to Python web scraping, covering the fundamental concepts of spiders, HTTP/HTTPS protocols, the Requests library usage, custom headers, proxies, cookies, and various data extraction techniques such as JSON parsing, XPath, and regular expressions.
This article reviews eight PHP web‑scraping libraries—Goutte, Simple HTML DOM, htmlSQL, cURL, Request, HTTPful, Buzz, and Guzzle—detailing their features, requirements, licensing, and documentation to help developers choose the right tool for their backend data‑extraction projects.
This guide introduces Scrapy, a powerful Python web‑scraping framework, explains its architecture and components, walks through installation, project creation, spider development, query syntax, recursive crawling, and item pipelines, providing practical code examples for building robust crawlers.
This article explains how to build a self‑healing dynamic proxy pool for 24/7 web crawling using Scrapy and Redis, covering requirements, design, implementation details, deployment steps, and a reusable Scrapy middleware example.
This guide explains practical techniques to avoid IP bans and 403 errors when web‑scraping, covering explicit and implicit waiting, User‑Agent spoofing, proxy usage, IP pools, and distributed crawling architectures.
When the Chinese railway ticketing system 12306 crashes under heavy load, developers turn to open‑source Python bots that simulate user behavior, query seat availability, and automate order submission, with detailed feature lists, repository links, and real‑world log examples.
This tutorial walks you through planning, extracting, and saving classic Chinese novel content from shicimingju.com using Python, regular expressions, and file storage, providing clear code examples and practical tips for successful web scraping.
This guide introduces Python’s Requests library, covering installation, GET and POST requests, handling headers, response codes, cookies, redirects, timeouts, and proxy settings, with practical code examples to help developers perform reliable HTTP operations and avoid common pitfalls in web scraping.
This article explains the legal boundaries of web crawling in China, covering recent cybersecurity regulations, what makes a crawler illegal or legal, common developer questions, and practical advice to avoid personal‑data violations and criminal liability.
This article explains how to scrape Chinese Basketball Association (CBA) match data from a portal website, analyzes the page structure, extracts table rows using regular expressions, converts them to CSV format, and provides a complete Java/Groovy code example for automated data collection.
This article presents seven practical Python scripts—from scraping Zhihu images and chatting bots to poetry author detection, lottery generation, auto‑drafting apologies, screen recording, and GIF creation—showcasing how to quickly automate diverse tasks without reinventing the wheel.
This article explains how to use a Groovy script to crawl over 7 million historical weather entries for 3,200 cities spanning 2011‑2019, process the JSON responses, and store the cleaned data into a MySQL table, while sharing practical tips and code snippets.
This guide explains practical techniques such as faking User‑Agent headers, rotating cookies, adding random delays, and using proxy pools to prevent IP bans while crawling large amounts of data from websites with anti‑scraping defenses.
This guide walks through using Python's requests and BeautifulSoup libraries to simulate Zhihu login, extract user profiles, retrieve answer likers, download avatars, fetch all answers for a question, and store the collected data in a SQLite database.
This tutorial walks you through using Python to collect tourism data from Qunar, extract key fields such as name, price, and rating, store the results in Excel with pandas, and visualize sales and popularity trends using pyecharts, including a simple recommendation algorithm.
This article introduces Puppeteer, a Node library for controlling Chrome, and demonstrates its use in automating web performance analysis and scraping tasks.
This article demonstrates how to use Selenium and Python to crawl over 6,000 Chinese tourist attractions from Qunar, extract ratings, popularity and sales data, and visualize the results with pandas, seaborn, matplotlib, and pyecharts, revealing the most visited sites and regional travel trends during the 2019 National Day holiday.
This article walks through the challenges of scraping Dazhong Dianping, explains how the site hides numeric data with custom CSS fonts, and provides a complete Python workflow—including HTTP requests, font extraction, glyph rendering, and OCR—to decode and retrieve the protected information.
This guide explains core XPath concepts—including node selection shortcuts, predicate filters, wildcard usage, multiple path unions, axis navigation, and built‑in functions—providing clear examples for effective XML/HTML data extraction.
After uncovering an unprotected API that allowed unlimited resource access, the author created a rough Java program that uses a fixed-size thread pool and CountDownLatch to fetch 100 000 items in parallel, retrieving 10 000 records per thread via HTTP GET requests.
The article walks through a Java solution for extracting multilingual data from an HTML table using regular expressions, handling spacing and encoding issues, splitting fields, and constructing INSERT statements to populate a country_code database table.
This tutorial walks you through logging into Taobao with Python requests, handling anti‑scraping measures, extracting product information via the PC search API, parsing JSON data, and saving the results to Excel, while also covering common pitfalls like sliders and proxy management.
Learn a step‑by‑step Python solution to locate, download, and store high‑resolution, royalty‑free images from ColorHub by navigating its three‑tier page structure, generating request headers, parsing HTML with BeautifulSoup, and saving files locally, enabling offline PPT creation without copyright concerns.
This tutorial introduces BeautifulSoup, a powerful Python library for parsing HTML and XML, covering installation, basic usage, tag selection, attribute extraction, navigation of parent and sibling nodes, method and CSS selectors, and best‑practice recommendations for efficient web data extraction.
