This tutorial shows how to crawl the China Meteorological Administration website to obtain today's weather and a six‑day forecast, handle Chinese encoding, extract data with XPath, and save the results into a CSV file using Python's requests and lxml libraries.
This tutorial demonstrates how to use Python on Windows to crawl Maoyan movie rankings, extract details such as title, rating, genre, region, and duration, store them in a CSV file, and then perform comprehensive data cleaning, analysis, and visualization with pandas, matplotlib, and WordCloud.
This tutorial demonstrates how to use Selenium to crawl short comments from a Douban movie page, extract them via XPath, paginate through results, and finally create a visual word cloud using Python's wordcloud library.
This article demonstrates how to use Python's requests and BeautifulSoup to crawl text from a news site, process it with collections, numpy, and jieba for word‑frequency analysis, and then visualize the top terms using pyecharts, providing complete code snippets and explanations.
This article explains a complete Python workflow for solving image‑click captchas by retrieving the base64 image, binarizing it, detecting character contours, generating candidate character images, matching them, and finally returning the click coordinates needed for verification.
This tutorial explains JSON fundamentals, demonstrates Python's json module functions, introduces the JSONPath library for extracting data from JSON documents, and provides a complete example of crawling city information from a website and parsing it with JSONPath.
This article demonstrates how to use Python's Selenium WebDriver to log into Alibaba International, scrape company contact information across multiple pages, save the data to CSV, download product images, and embed them into an Excel file, providing a complete automation workflow.
This guide demonstrates how to use Python on macOS to scrape love quotes, store them, and automatically launch QQ to send a random quote each day, including code snippets and steps to set up an Automator app for startup execution.
This article demonstrates step‑by‑step how to overcome a website’s anti‑scraping defenses using Selenium with stealth options, retrieve CSS‑based price images, reconstruct the digits, and apply Tesseract OCR to accurately extract numeric data, providing complete Python code snippets throughout.
This tutorial demonstrates how to crawl approximately 40,000 Maoyan movie reviews for the film “Chinese Doctors,” preprocess the data, and create visualizations such as rating pie charts, city distribution maps, top‑viewer bar charts, and a word cloud using Python libraries like requests, pyecharts, and wordcloud.
This article explains how font‑based anti‑scraping works, shows how to locate and download custom font files, decode their glyph mappings into a dictionary, and use the mappings to extract real data from a recruitment site with Python, Scrapy and MySQL.
Learn step‑by‑step how to fetch Baidu Index search volume data for any keyword using Python: discover the API endpoints, decrypt the encrypted response with a custom function, and wrap the process into reusable code without relying on UI automation tools.
This article demonstrates a simpler method to obtain water‑marked‑free Douyin video URLs by inspecting network requests, extracting hidden links, and using a Python script with the jsonpath library to download videos directly, highlighting advantages and remaining limitations.
This tutorial demonstrates how to analyze an e‑commerce search page, use Selenium to automate browsing, extract product details across multiple pages, and store the collected data into a MySQL database, providing complete Python code and step‑by‑step explanations.
This guide walks through using Python's requests and lxml libraries to scrape quotes, authors, and tags from quotes.toscrape.com, parse the HTML with XPath, and store the results in a CSV file, covering request headers, element inspection, and data extraction steps.
This article introduces Python 3 methods for retrieving various document types—including TXT, CSV, PDF, and Word files—using urllib, regular expressions, and file‑specific processing steps, providing practical code examples and workflow guidance for building effective web crawlers.
This tutorial explains how to create a Python Selenium script that automatically opens Taobao, logs in via QR code, navigates to the shopping cart, waits for a preset flash‑sale time, selects all items, and completes the checkout process.
This tutorial shows how to build a Python web‑scraper that extracts book titles and authors from a website, formats the data, and automatically sends it via email on a scheduled interval using SMTP and the schedule library.
This article walks through the step-by-step process of extracting individual and batch stock minute-level data from Eastmoney using Python, covering URL analysis, parameter discovery, handling pagination, constructing CSV output, and extending the script to fetch arbitrary dates via cookie manipulation.
Learn step-by-step how to collect worldwide electricity generation data from the IEA using Python's requests and pandas, extract country names, retrieve historical consumption figures, and turn the results into animated visualizations with online tools, while avoiding manual entry and ensuring reproducible analysis.
This tutorial walks you through using Python's built‑in Tkinter library to build a small GUI application that fetches personalized signature images from an online service, downloads them, and displays the result, complete with code examples and UI design tips.
This article demonstrates how to use Python to scrape fund data from Eastmoney, store it in Excel, visualize fund type distribution, and perform detailed time‑series analysis on a specific fund's net‑value, growth rate, and yearly performance.
This tutorial demonstrates how to use Python to scrape Weibo for the most common National Day leave reasons, extract comment data via Ajax requests, store it in CSV files, and create visualizations such as top‑liked users and word‑clouds to reveal popular motifs.
This article walks through a real‑world Python project that uses Selenium to crawl the first ten pages of Taobao snack listings, extracts sales, price and location data, visualizes price distribution and geographic concentration, generates a word‑cloud of top user comments, and lists the top‑selling stores, providing full source code for replication.
Learn step‑by‑step how to crawl 46,000+ rental listings from Ziroom using Python, extract house details with regex, clean and transform the data with pandas, and visualize distribution, pricing and location insights through pyecharts, matplotlib and seaborn, revealing rental market patterns in Beijing.
This tutorial shows how to use Python's requests and JSON handling to crawl the Shanghai Stock Exchange website, extract periodic report metadata, construct PDF URLs, and automatically download the files, providing a practical example of backend web‑scraping automation while warning against excessive server load.
This article walks you through using Scrapy to fetch price data from Beijing Xinfadi's website, covering request analysis, spider creation, item definition, pagination, data extraction, pipeline setup, and exporting results to CSV with full code examples.
This tutorial demonstrates how to use Python's requests and lxml libraries to scrape static web pages, extract image URLs via XPath, convert thumbnail links to full‑size URLs, and download the images to a local folder, providing a complete end‑to‑end example.
This article walks you through using Python's Selenium library to programmatically retrieve Bilibili video series playlists, explains the required XPath selectors, provides a complete, runnable code example, and addresses common issues such as ChromeDriver compatibility.
This article explains how to create a Python web crawler that automatically generates URLs for a drama‑download site, filters out invalid pages, extracts ed2k links using requests and regular expressions, saves them to text files, and employs multithreading to speed up processing, while discussing challenges such as duplicate URLs and filename sanitization.
This article presents a Python‑based web‑scraping project that collects the first ten pages of "小鱼零食" search results on Taobao, analyzes sales, price distribution, geographic store locations, top comments, and visualizes the findings with bar charts and word clouds.
This tutorial walks you through analyzing Tencent's recruitment page, locating the Ajax JSON endpoint, and using Scrapy to create a project, spider, items, pagination, settings, and data export to collect job postings efficiently.
This article explains how to scrape tens of thousands of images from Sogou by analyzing the request URLs, extracting image URLs from JSON responses, and implementing a multithreaded Java downloader that stores the pictures locally.
This tutorial explains how to collect captcha images, preprocess them with OpenCV—including grayscale conversion, median blur, binarization, contour detection, and character segmentation—and provides core Python code and visual results for building a reliable Python web‑scraping pipeline.
This tutorial shows how to use Python to fetch the Weibo hot search list, clean and organize the data with pandas, send the results via QQ email, and schedule the whole process to run automatically each evening.
This tutorial walks you through using Python to collect JD.com product comments, parse the JSON responses, store the data in Excel, and then perform simple visual analysis such as color‑distribution bar charts and a word‑cloud of review content.
This tutorial explains how to install and use the Python jsonpath library to efficiently query and extract values from deeply nested JSON structures, covering basic syntax, indexing, conditional filtering, and result formatting with clear code examples and illustrations.
This article walks you through cleaning messy Taobao product data using pandas and jieba, then visualizing ingredient and shelf‑life information with interactive Pyecharts charts—including pie, bar, table, funnel, and polar graphs—while showing how to combine multiple charts into a single draggable HTML page.
This article provides a comprehensive, step‑by‑step guide to the Scrapy web‑crawling framework, covering its core components, installation methods, project layout, spider creation, data extraction techniques, pagination handling, pipeline configuration, and how to run the crawler to collect and store data.
This tutorial demonstrates how to capture network packets from the Super Schedule Android app, perform a login request with required headers, and repeatedly fetch and parse topic data using Python urllib2 and JSON handling, enabling infinite scrolling of user‑generated topics.
This guide explains how to analyze Sogou's image search XHR request, extract image URLs from the JSON response, and use Java with multithreaded HTTP requests to download thousands of pictures efficiently, including full source code and execution tips.
This tutorial walks through creating a stock sentiment analysis pipeline in Python by scraping news from a financial website, applying Baidu AI's sentiment API to classify headlines, aggregating positive and negative ratios per stock, and visualizing the results with matplotlib.
This article demonstrates how to crawl Bilibili creator data, analyze fan counts, categories, gender and video statistics with Python and pandas, and then create a nickname generator for aspiring million‑follower up‑hosts using both Python and JavaScript.
This tutorial walks through collecting complaint data from the Black Cat platform during the pandemic, cleaning and merging the datasets with pandas, analyzing temporal trends, and generating insightful word clouds using jieba and WordCloud, all illustrated with Python code and visualizations.
This article demonstrates how to scrape attraction and food data for Changsha using Python, process it with pandas, visualize insights with Plotly and Pyecharts, and derive travel recommendations such as top scenic spots, popular eateries, and price trends.
This tutorial walks you through creating a feature‑rich desktop music player in Python using PyQt5, covering UI layout design, keyword‑based music crawling, multithreaded downloading, playback controls, volume adjustment, and additional functions like random and repeat modes, all illustrated with complete code snippets and screenshots.
This article walks you through building a Python web‑scraper that extracts second‑hand housing data from Lianjia, covering target identification, page structure analysis, multi‑level filtering, pagination handling, detailed page parsing, and data storage with practical code examples.
This guide explains how to programmatically log into a website using Python's urllib2 and cookielib modules, covering cookie handling, captcha retrieval, form data construction, request headers, and posting credentials to access protected pages.
This article walks through a Python-driven web‑scraping and data‑analysis workflow that collects, cleans, and visualizes tourism and food data for Changsha, revealing the most popular sights, highly‑rated eateries, and visitor preferences through interactive charts and word clouds.
This tutorial walks you through analyzing a JavaScript‑driven website, discovering the hidden API, configuring request headers with Postman, and writing a Python scraper that extracts book titles, authors, codes, and prices from epubit.com.
This article explains how to overcome common anti‑scraping defenses such as identity verification and IP rate limiting by spoofing the User‑Agent header and adding request delays, providing complete Python code examples using requests and BeautifulSoup to scrape Douban's Top 250 movies.
This comprehensive guide introduces Scrapy's architecture, explains its core components and data flow, teaches XPath fundamentals, walks through installation, project creation, spider coding, item and pipeline definitions, middleware customization, pagination handling, and essential settings for effective Python web crawling.
Learn how to automate a web-based Sudoku game using Python's Selenium for data extraction, implement an efficient backtracking and bitwise algorithm to solve the puzzle, and programmatically fill the solution back into the browser, achieving performance improvements from 17 seconds to just over 3 seconds.
This guide walks you through analyzing Toutiao’s Ajax requests, extracting image URLs, and using Python’s requests library to download street‑photo pictures in bulk, complete with code snippets and tips for pagination and future enhancements.
This guide explains how to combine Scrapy and Selenium to scrape dynamically rendered web pages, covering installation, project setup, middleware configuration, Selenium driver handling, and code examples that demonstrate a complete end‑to‑end crawling workflow.
This tutorial walks you through installing Selenium and ChromeDriver, configuring the environment, and using Python code to automatically navigate Dangdang's bestseller pages, extract book details with pyquery, and save the results into a CSV file for further analysis.
This guide walks you through building a Python web‑scraper that extracts NBA player information from the Hupu website, cleans and visualizes the data, and automatically creates Excel files with embedded line charts, covering URL navigation, data parsing with requests and BeautifulSoup, and chart generation with xlsxwriter.
This article presents seven practical Python scripts—from a concise web scraper for Zhihu images and a chatbot conversation loop to a Naive Bayes poem author classifier, a lottery number generator, an automated essay writer, a screen‑capture tool, and a GIF creator—demonstrating how to avoid reinventing the wheel while exploring diverse automation tasks.
This article demonstrates how websites can detect browsers driven by Selenium or Puppeteer, walks through experiments using bot.sannysoft.com, shows screenshots of detected features in both headed and headless modes, and explains why such tools often fail to evade detection.
This tutorial walks beginners through the complete process of building a Python web scraper to collect internship listings from Shixi.com, covering page analysis, static vs dynamic detection, data location with XPath, pagination handling, and storing results in a CSV file using pandas.
Learn step-by-step how to use Python's Selenium library to automate login, scroll through QQ Space, extract historical posts and download album photos, including driver setup, code snippets for login, shuoshuo retrieval, and photo downloading, enabling you to preserve nostalgic content.
This article shares seven practical Python scripts—from web‑scraping Zhihu images and chatting bots to AI poem analysis, lottery generation, automatic apology letters, screen recording, and GIF creation—demonstrating how to avoid reinventing the wheel while learning useful automation techniques.
This guide explains why Python is essential, presents a step‑by‑step learning roadmap covering beginner basics, backend web development, web crawling, data analysis, and machine learning, and provides curated resources and project links to help learners progress efficiently.
The article presents a comprehensive guide to Python's built‑in functions, standard libraries, and third‑party packages across file I/O, web scraping, databases, data cleaning, statistical analysis, machine learning, visualization, and more, rating each with stars and offering a free e‑book collection for readers.
This tutorial guides readers through analyzing web pages, using Python to write network programs, applying regular expressions, and operating Excel to scrape all listed company stock data for a specified date range, save it by stock code, and optionally store results in databases.
This article provides a comprehensive Python web‑scraping guide covering basic GET/POST requests with urllib2, proxy handling, cookie management, header manipulation to mimic browsers, gzip compression handling, regular‑expression and library parsing, simple captcha strategies, and a multithreaded thread‑pool example.
This article explains how to extract song information from KuGou by inspecting network requests, simplifying URLs, generating required MD5 signatures, constructing authenticated API calls, and using Python to download the resulting audio files to a local directory.
This article presents a practical collection of Python regular‑expression techniques for extracting HTML elements such as table rows, links, titles, images, and scripts, showing how to filter tags and handle URL parameters during web crawling.
This guide demonstrates how to use Python's Selenium library to log into a GPS web system, locate elements via name and XPath, handle iframes, click list items, and extract dynamic data, including detailed installation steps for Selenium and ChromeDriver.
This tutorial demonstrates how to use Python's requests library to simulate Zhihu login by handling dynamic _xsrf tokens, optional captcha verification, saving cookies, and then crawling the main page to extract questions and answer abstracts.
This article introduces the BeautifulSoup library, explains how to install it, demonstrates core parsing methods such as find, find_all, select, and relationship navigation, and provides a complete example of scraping novel titles from Qidian using Python requests.
This step‑by‑step guide shows how to log into Toutiao, locate the fan list via browser dev tools, capture the AJAX request URLs, decode Unicode nicknames, paginate through all pages, and save the results as JSON using Python.
This tutorial walks through analyzing a Qidian novel page, extracting GET request parameters, constructing the catalog URL, and repeatedly fetching free chapters with Python, handling asynchronous loading and delays to reliably download the novel’s free content.
This tutorial shows how to create a multithreaded Python web scraper that fetches news from Sina and NetEase, stores the results in MySQL, generates a live word‑cloud, and serves the data through a simple Flask web application with an HTML front‑end.
This article introduces three Python libraries—PyExecJS, Js2Py, and PyV8—that enable execution of JavaScript within Python, showing how to set up runtimes, run code snippets, read JS files, and scrape encrypted web pages for more effective data extraction.
This article walks through using Python's requests and lxml libraries to fetch, parse, and download songs from NetEase Cloud Music, detailing the implementation steps, code snippets, encountered obstacles, and tips for handling anti‑scraping measures.
This article walks through building a Python web crawler that extracts forum post comments into MongoDB, covering project goals, environment setup, site structure analysis, Scrapy and Selenium integration, data storage design, handling anti‑scraping measures, and performance optimization with multithreading.
This guide explains how to locate the JavaScript responsible for generating URL parameters by inspecting event listeners and searching source files in Chrome, observe its execution with breakpoints, and then replicate the logic in Python using js2py to obtain login credentials.
This article demonstrates a Groovy script that extracts QR‑code image URLs from a web page using regular expressions, then downloads each image to a local directory, illustrating practical web‑scraping techniques and reusable utility methods for HTTP requests and file handling.
InfoSpider is an open‑source Python‑based web‑scraping toolbox that securely aggregates personal data from over 24 sources—including email, e‑commerce, and social platforms—provides a GUI for easy operation, stores results in JSON, and offers basic visual analysis, making personal data integration and insight generation straightforward.
This article walks you through building a Python web‑scraper that queries Baidu for the keyword “grain,” extracts article titles, abstracts and links using Requests and lxml, saves the results to a text file, and shows the execution output with screenshots.
This article demonstrates how to build a Java-based web crawler that fetches image URLs from Sogou Image Search, parses the JSON response, and downloads thousands of pictures to a local directory using multithreaded processing and custom HTTP utilities.
This article provides a comprehensive guide to Python web scraping, covering basic GET/POST requests, proxy usage, cookie management, browser header spoofing, gzip compression handling, and multithreaded crawling to improve efficiency and avoid common obstacles.
This guide walks you through configuring PyCharm to connect to a remote Linux server, set up an SSH Python interpreter, enable automatic file synchronization, and run long‑running web‑scraping scripts directly on the server, streamlining development without manual file transfers.
Learn step‑by‑step how to scrape Tencent’s location data via its API, process the coordinates with pandas, deduplicate and filter them, and finally visualize regional foot‑traffic as an interactive heatmap using Folium, complete with code snippets and practical tips.
Using Python, this tutorial demonstrates how to scrape 1,585 cherry product listings from Taobao, extract key fields, clean and transform the data, and visualize regional sales and price distributions, highlighting the most popular provinces, stores, and price ranges, all with reproducible code.
This article introduces Microsoft’s open‑source Playwright‑Python library, explains its advantages over Selenium, walks through installation, code generation, synchronous and asynchronous usage, mobile emulation, and highlights its strengths and current limitations for Python developers.
In this article we demonstrate how to use Python’s requests and BeautifulSoup to scrape a static webpage listing the world’s top 100 most beautiful women, process the data with pandas and pyecharts for continent, nationality and profession statistics, and apply Baidu’s face‑recognition API to score each celebrity’s attractiveness.
This tutorial walks you through installing Scrapy on Windows, creating a project and spider, configuring settings, and using XPath to crawl and extract joke titles and contents from the Qiushibaike website, providing a solid foundation for Python web scraping.
This guide demonstrates how to retrieve a list of WeChat public account articles by constructing the appropriate API URL, examining its query parameters, viewing the JSON response, and using a Python script to parse the data and generate a markdown file of titles and links.
This article demonstrates how to use Python's requests, fake_useragent, and lxml libraries to batch‑scrape nearly a thousand new‑house listings from the 惠民之家 website, extracting 41 fields such as name, price, layout, opening date, plot ratio and green ratio, while handling pagination and anti‑scraping measures.
This tutorial explains how to use Python's requests, lxml, and urllib libraries to search Baidu Tieba by keyword, bypass anti‑crawling measures, extract image and video URLs with XPath, and save the media files locally, complete with code examples and screenshots.
This article introduces the fundamentals of XPath syntax, common rules, and example expressions, then explains how to use the lxml library in Python for HTML/XML parsing, including practical tips and a complete code example for extracting links and text from a sample document.
This tutorial shows how to use Python to fetch data from Bilibili’s “马保国” channel via its public API, extract video metadata, clean and visualize 14,000 records, and generate insights such as top‑viewed videos and a comment word cloud.
Learn how to build a Python web scraper that extracts iQIYI movie titles, actors, and scores, parses the data with regex or BeautifulSoup, displays results in a Tkinter GUI with a combobox, and saves the information to a file—all explained with code snippets and screenshots.
This article demonstrates how to crawl 6,025 Beijing Dank Apartment listings using Python, clean and enrich the data with Pandas, and visualize distribution, price, size, floor, and subway proximity through charts, revealing key market insights and correlation patterns.
This article explains how to locate Weibo comment APIs, work around rate limits by using the mobile site, extract required parameters, and implement a Python script that handles cookies, pagination, emoji cleaning, deduplication, and scheduled execution to collect comments efficiently.
This article provides a step‑by‑step guide on using Scrapy’s crawl template to automatically scrape product details such as ID, title, shop name, shop link, and price from JD.com, including source analysis, project setup, code snippets, and result verification.
This article walks through the complete process of designing a Python web scraper to collect Zhihu answer data, parse author, ID and excerpt fields, store the results in CSV files, and finally visualize the text with a word‑cloud, including all necessary code snippets and explanations.
This article walks through using Python to crawl Tencent Video's "Offer" season 2 comments, merge and clean the CSV data, perform exploratory analysis, generate visualizations and word clouds, and apply Baidu's open‑source NLP model for sentiment scoring, providing complete code snippets for each step.
