This tutorial explains how to analyze the Laifeng live‑stream chat API, extract the required timestamp and signature parameters from JavaScript, and implement a Python script that logs in, generates the encrypted sign, and sends custom danmu messages to a specified room.
This tutorial demonstrates how to scrape Maoyan movie comments for "The Battle at Lake Changjin", clean and store the data, perform comprehensive visual analyses such as likes, city, gender, watch status, rating, user level, and creator mentions, and finally predict next‑day box office using linear regression with sklearn.
This tutorial demonstrates how to use Python's requests and lxml libraries to scrape quotes, authors, and tags from quotes.toscrape.com, parse the HTML with XPath, and save the extracted data into a CSV file.
This article explains why Chinese characters often appear as garbled text when crawling websites with Python and provides three practical solutions—using response.content, manually setting the response encoding, and applying a generic encode‑decode trick—to reliably decode the data.
The guide explains how to use Python libraries such as requests, pandas, lxml, and regex to automatically paginate through a website, extract PDF file names and URLs, create organized folders, and download all files in bulk with minimal manual effort.
This article demonstrates how to build a Python web scraper using requests, lxml, regular expressions, and tqdm to locate, extract, and download video files from a VIP anime website, covering header configuration, XPath parsing, URL reconstruction, and file saving.
This tutorial explains how to use Python's requests and BeautifulSoup libraries to fetch a webpage, parse its anchor tags, filter and construct download URLs, and automatically download multiple MP3 files by writing the binary content to local files.
This article describes how to build a Python web crawler that automatically generates numeric URLs, checks their validity, extracts download links for TV dramas from cn163.net, saves them to text files, and discusses practical challenges such as regex parsing, filename restrictions, and multithreading performance.
This tutorial walks you through using Python Selenium to automate QQ Music, scroll through the infinite‑scroll comment section, extract user avatars, names, timestamps and comment text, save the data to CSV, and finally visualize the comments with a word‑cloud image.
This tutorial shows how to use Python to scrape GIF images, automatically generate romantic messages, obtain AI‑generated replies, and send the whole package as an email through a 163 SMTP server, providing a complete, runnable example.
This tutorial walks through using Python's Selenium to automate scrolling and extract QQ Music hot comment data—including user avatars, names, timestamps, and content—then saves the information to CSV and creates a Chinese word cloud for visual analysis.
This tutorial walks through designing and implementing a Python desktop music player using PyQt5, covering UI layout, keyword‑based music crawling, multithreaded downloading, playback controls, volume adjustment, and various playback modes such as sequential, random, and repeat.
This article walks through creating a full‑featured desktop music player using Python 3.6, PyQt5 for the GUI, and requests for web scraping, covering environment setup, dependencies, key functionalities, technical challenges, and a reflective conclusion on the development experience.
This tutorial walks you through using Python's requests library to inspect Kugou Music's web requests, simulate them, extract MP3 URLs, and download songs, covering preparation, goal setting, implementation steps, and a concise project summary.
This tutorial demonstrates how to programmatically log into a website by handling cookies, fetching and solving captchas, and posting the required form data using Python's urllib2 and cookielib modules, with a complete example targeting a school academic system.
This tutorial shows how to use Python libraries such as requests, lxml, pandas and pyecharts to crawl the latest COVID‑19 statistics from Baidu, store them in an Excel file, and create interactive province‑level maps that illustrate confirmed, death and recovery numbers.
This tutorial explains how to use XPath in Python to extract free proxy IP addresses from kuaidaili.com, covering XPath basics, installing the XPath Helper Chrome extension, crafting extraction rules, and looping through pages to collect and display results.
This article demonstrates how to use Python to scrape historical lottery data, apply a custom pattern‑matching algorithm to identify winning opportunities, aggregate the results over multiple days, and export the analysis to an Excel file for further review.
This article presents a collection of Python 3.6.4 scripts that demonstrate how to scrape Zhihu images, converse with chatbots, classify Tang poems using NLP, generate random lottery numbers, automatically compose apology letters, capture screen images, and create animated GIFs, providing practical code examples for each task.
This tutorial walks through building a Python web crawler using Scrapy and Selenium to extract forum comments, store them in MongoDB, handle anti‑scraping measures, avoid duplicate data, and demonstrates the full end‑to‑end process with code examples and results.
This tutorial demonstrates how to use Python's urllib module to simulate browser requests, parse HTML with Beautiful Soup, extract text and image URLs, and store the scraped data locally using file I/O and the with statement, providing complete code examples.
This article explains how to use Python's requests and lxml libraries to scrape NetEase Cloud Music's song list, extract song IDs, construct download URLs, and save the audio files locally, while also discussing common obstacles such as anti‑scraping measures and network errors.
This tutorial explains how Selenium can be used to simulate browsers for scraping JavaScript‑rendered pages, covering browser selection, page navigation, element locating methods, interaction techniques, action chains, JavaScript execution, frame handling, waiting mechanisms, navigation controls, cookie management, and tab management.
This tutorial demonstrates how to use Python, requests, and openpyxl to analyze a job‑listing website, construct the appropriate Ajax URLs, write a scraper that parses JSON responses, and export the collected position data into an Excel workbook.
This tutorial explains how to analyze the Ant Short‑Term Rental website's anti‑scraping mechanisms, extract the required Cookie and User‑Agent headers, and use Python's urllib2 and BeautifulSoup to reliably crawl rental listings, save the data to CSV, and optionally extend the scraper with Selenium.
This tutorial demonstrates how to use Python's requests and BeautifulSoup libraries to scrape the titles and release years of the top 250 movies from Douban, clean the data, export it to Excel, and then create pivot tables and charts to visualize yearly film production trends.
This article introduces Python's versatile ecosystem, highlighting eight key tools—including IDLE, scikit-learn, Theano, Selenium, TestComplete, Beautiful Soup, Pandas, and PuLP—that empower developers in IDE usage, machine learning, web automation, data analysis, and optimization tasks.
This article explains why web servers block crawlers, how to identify a browser's User-Agent (using Chrome as an example), and demonstrates step‑by‑step how to disguise a scraper as a browser to retrieve the CSDN homepage and its article list.
This article explains how to build a Python web scraper that logs into a site with Selenium, extracts HTML content, parses it with BeautifulSoup and html5lib, and processes table data, while also covering anti‑scraping techniques and required libraries.
This article demonstrates how to crawl Douban short reviews for the TV show "Actors Please Take Your Place" Season 2, clean and deduplicate the data, apply Baidu's SKEP sentiment model, and visualize word clouds, rating distributions, posting times, and sentiment scores, providing full Python code for replication.
This article explains how to capture a video site's POST data with Fiddler, locate encrypted M3U8 stream URLs, and use Python 3 with the pycryptodome library to decrypt and download the video files for playback or conversion.
This guide walks through analyzing a Tenda router's web interface to discover its data‑fetching URLs, extracting all available modules, and then using Python's requests library to retrieve the information and expose it through a simple Flask API, with both object‑oriented and functional code examples.
This tutorial explains how to use Python's requests and lxml libraries to crawl a website, bypass SSL verification, extract audio file URLs, and download sound effect MP3s in bulk, providing complete code snippets and practical tips for safe web scraping.
This tutorial walks you through using Python's requests and fake_useragent libraries to locate Sogou wallpaper JSON endpoints, extract image URLs, bypass basic anti‑scraping measures, and download the pictures to a local folder, complete with full code examples and explanations.
This tutorial explains how to build a Python web scraper using Selenium and Beautiful Soup, covering setup, login automation, HTML extraction, data parsing with html5lib, handling anti‑scraping measures, and best practices for large‑scale data collection.
The article explains how using an element's string representation as a cache key caused duplicate extraction in a Python news‑page parser, reveals the role of Python's garbage collection and memory reuse, and shows how switching to XPath keys resolves the bug.
This article demonstrates how to use Python to scrape Meituan for Shenzhen barbecue restaurant data, clean and enrich the dataset with Pandas, and visualize market insights such as regional distribution, average spending, ratings, and shop types using pyecharts and Matplotlib.
This article explains how to use Python and MitmProxy to automatically capture reading counts, likes, and view numbers from WeChat public articles, covering installation, dynamic cookie/token handling, batch fetching, and full code examples for a streamlined backend scraping solution.
This article explains a puzzling bug in the General News Extractor where cached HTML elements produce inconsistent results, analyzes how Python's garbage collection and memory reuse cause duplicate text extraction, and shows how switching the cache key to XPath resolves the issue.
This tutorial explains how to use Python's requests, lxml, and fake_useragent libraries to crawl the free novel website "无限小说网", extract each novel's download link, and automatically download the corresponding text files.
This tutorial walks through building a Python web scraper that fetches Douban TV show titles, ratings, detail links, and images, parses JSON responses, handles anti‑scraping measures, and stores the results in a CSV file with step‑by‑step code examples.
This tutorial walks you through using Python's requests, lxml, and fake_useragent libraries to crawl Qyer travel city pages, extract city names, images, and hotspots, and store the results in a CSV file while handling pagination and request headers.
This tutorial demonstrates how to use Python's requests, lxml, and pprint libraries to crawl the Hao123 travel website, extract attraction names, opening times, reviews, and prices, and display the results, while providing complete code examples and practical tips.
This tutorial walks you through using Python's requests and lxml libraries to reverse‑engineer the Youdao mobile translation interface, construct the required form parameters, send POST requests, parse the returned HTML with XPath, and display translated results for multiple languages.
This tutorial explains how to create a Python CLI application named tickets that parses command-line arguments, retrieves station codes, queries the 12306 railway API for ticket data, and displays the results in a formatted table using libraries such as docopt, requests, and prettytable.
This article demonstrates how to employ Pandas' read_html function to quickly fetch, parse, and analyze the Wikipedia table of the world's richest billionaires, covering basic usage, ranking by net worth, selective column extraction, and advanced parameter options.
This article demonstrates how to build a Python web scraper for Toutiao that leverages a multiprocessing pool, identifies AJAX endpoints, fetches JSON data, extracts titles and image URLs, handles Unicode encoding, and downloads images into organized folders.
This tutorial explains how to use Python's requests and lxml libraries to crawl a public image site, parse pagination URLs, retrieve high‑resolution 王者荣耀 pictures, and save them locally in batches, while highlighting key implementation steps and best‑practice cautions.
This guide walks you through setting up Python, installing required libraries, capturing Taobao product URLs, logging in, parsing review data with BeautifulSoup, and saving the results, while highlighting best practices to avoid overloading the server.
This tutorial explains how to automate bulk image downloads from a website using Python's requests, lxml, and fake_useragent libraries, covering environment setup, header configuration, URL formatting, page iteration, HTML parsing, and download logic in a clear, practical example.
This article introduces Python's core concepts and hierarchy, highlights ten frequent beginner mistakes, and walks through building a basic web scraper that extracts book information from Douban, processes it with pandas, and displays the resulting data, providing a practical learning path for Python fundamentals.
This tutorial walks you through creating a Python-powered QQ Music scraper that retrieves song details, lyrics, and comments, generates a word‑cloud visualization, and packages the functionality into a menu‑driven command‑line tool, complete with code snippets and troubleshooting tips.
This article walks through using Python to download all WeChat group member avatars, save them locally, and then apply Baidu's face‑recognition API to evaluate age, gender and beauty scores, finally filtering and ranking the images based on the results.
This article explains how to use Python's urllib2 library for web scraping, covering basic page fetching, proxy configuration, cookie handling, form submission, header manipulation, anti‑hotlink techniques, and advanced methods like Selenium to bypass restrictions.
This article demonstrates how to use Python's requests and regular‑expression libraries to crawl Taobao product listings, extract product titles and prices, handle pagination, and store the results, providing complete sample code for each step.
This comprehensive guide walks you through Python web crawling fundamentals, covering request handling, URL encoding, regular expressions, the requests library, XPath parsing, and lxml, complete with code snippets and practical examples to help you build effective scrapers.
This guide introduces the most important Python libraries for web scraping—including requests, urllib3, Selenium, aiohttp, BeautifulSoup, lxml, pyquery, PyMySQL, PyMongo, and redisdump—explaining their core features, typical use cases, and providing concise code examples to help beginners get started quickly.
This tutorial walks through analyzing a website’s image categories, constructing parameterized URLs, creating a Tkinter GUI to select download paths and categories, handling filename issues, and finally packaging the Python scraper as a standalone executable.
The author presents a Java HttpClient crawler that efficiently retrieves over 60,000 nationwide school names in about 16 minutes by replacing slow Selenium UI scripts with direct API calls, storing the results in a database, and shares the complete source code for reference.
This tutorial shows how to log in to Douban with Python, scrape short comments for the animated film “Nezha”, process the text using jieba for Chinese word segmentation, and generate a visual word cloud that highlights the most frequent audience sentiments.
This tutorial walks you through extracting dynamic Maoyan review data for the hit animation "Ne Zha" using Python, parsing the JSON payload, storing it as JSON, and then visualizing ratings, gender distribution, city hotspots, and word‑cloud insights with pyecharts and wordcloud.
This guide explains why backing up WeChat public account articles is essential and provides a Python solution that captures article content, fixes image URLs, and generates PDF files—either one per article or a combined document—complete with code examples and usage tips.
This article explains how to crawl Zhihu's "beauty" topic images using Python, Requests, lxml, and Baidu's AipFace SDK, filter them by gender, face presence, realism, and beauty score, and store the high‑quality results locally.
This article describes how the author built a Python web crawler to automatically generate numeric URLs, fetch TV drama pages from the 天天美剧 site, extract ed2k download links using regular expressions, and save them into organized text files, streamlining the download process with Thunder.
This article demonstrates how to scrape the full 2018 Chinese variety‑show list from Douban using Python Selenium and BeautifulSoup, compile detailed metadata and actor information into Excel, and then analyze popularity rankings, rating distributions, frequent celebrity appearances, and common negative feedback.
This guide walks through eight practical Python urllib2 techniques for web crawling, covering basic GET/POST requests, proxy usage, cookie management, header spoofing, page parsing with regex and BeautifulSoup, captcha handling, gzip compression, and multithreaded fetching with a simple thread pool.
Learn step-by-step how to use Python’s itchat library to extract your WeChat contacts’ province and city information, handle QR code authentication, parse the data, and create visual maps of friend distribution, all illustrated with screenshots and code snippets.
This guide walks you through analyzing Meituan Takeout's comment API, constructing the correct AJAX request URLs, extracting review data with regular expressions, looping through pages, and storing the results in files or a database, complete with code snippets and screenshots.
This article presents seven practical Python projects—including a Zhihu image scraper, dual chatbot conversation, AI poetry author classifier, 35‑choose‑7 lottery generator, automatic apology letter writer, screen‑capture tool, and GIF creator—complete with ready‑to‑run code snippets and step‑by‑step explanations for developers eager to expand their programming repertoire.
This tutorial explains how to use Python's urllib module to fetch a Baidu Tieba page, apply regular expressions to extract image URLs, and then download and rename those images to your local directory.
This article explains how to create a Python web‑scraper that automatically generates URLs, fetches drama pages from a download site, extracts ed2k links with regular expressions, saves them to text files, and handles missing pages and filename restrictions efficiently.
This tutorial demonstrates how to use Python's itchat library to extract WeChat friends' signatures, save them to a file, and then generate a word‑cloud visualization with custom background and font, guiding readers through each step from data collection to visual output.
This article walks through the complete Scrapy implementation for extracting WeChat Moments data, covering item definition, spider configuration, request handling, parsing logic, pipeline setup, execution commands, and encoding fixes to produce a clean JSON output.
The article introduces the Requests‑HTML library—a powerful Python tool that merges HTTP downloading and HTML parsing, outlines its key features such as JavaScript support and built‑in selectors, and provides step‑by‑step code examples for quick, efficient web scraping.
This tutorial shows how to capture hot comments from NetEase Cloud Music using Python web‑scraping techniques, handle the required encryption parameters, and then transform the data into bar charts and word clouds with pyecharts and WordCloud for clear visual insight.
This guide walks through practical Python web‑scraping techniques using urllib2, covering basic GET/POST requests, proxy handling, cookie management, header spoofing, page parsing with regex/lxml/BeautifulSoup, simple captcha solutions, gzip compression, and a multithreaded crawling template.
This guide walks you through extracting Python job listings from Lagou.com with Python's requests library, parsing the JSON response, storing the results in CSV, and visualizing key insights such as education requirements, work experience, salary distribution, word clouds, and geographic salary heatmaps using pandas, matplotlib, and pyecharts.
This guide compiles essential Python snippets for tasks such as image manipulation with Pillow, video downloading via youtube‑dl, inspecting request objects, accessing NetEase Cloud Music, retrieving financial data with Tushare, running Docker‑based vulnerability labs, querying Beijing bus real‑time info, extracting articles with Goose, performing sentiment analysis, generating fake user agents, and more.
This tutorial explains how to collect the names and trading information of every stock listed on the Shanghai and Shenzhen exchanges using Python 3.5, requests, BeautifulSoup and regular expressions, then store the results in a file through a clear three‑step process of fetching the stock list, retrieving each stock's page, parsing the HTML and writing the data.
This tutorial explores Python metaclasses, demonstrating how to create dynamic classes, implement a lightweight ORM using metaclasses, and build a flexible proxy crawler with multiple site support, all illustrated with clear code examples and step‑by‑step explanations.
This article documents a Python‑based web‑scraping project that harvested over 1.38 million Zhihu followers, filtered high‑impact users, and visualized insights such as follower distribution, gender ratio, top influencers, geographic spread, education, industry, and certification details, highlighting challenges and lessons learned.
This tutorial walks through using Python 3.5 to fetch the latest movies from Douban, extract their IDs and titles, crawl user comments, clean the text with regular expressions, segment Chinese words using Jieba, remove stopwords, compute word frequencies, and finally generate a word‑cloud visualization of the reviews.
This article presents a curated list of 32 Python web‑scraping projects, each with a brief description of its purpose, technology stack, and data output format, helping developers quickly find useful open‑source crawlers on GitHub.
This tutorial walks you through creating a Python command‑line tool named “tickets” that, using virtualenv, docopt, and prettytable, retrieves real‑time train ticket information from China’s 12306 service by specifying departure, destination, and date, and displays the results in a formatted table.
This tutorial shows iOS developers how to set up Python on macOS, install pip and BeautifulSoup, fetch a web page with urllib2, parse the HTML to locate image tags, extract their URLs, and download the images programmatically for use in an app.
This article walks through building a Python web scraper for Zhihu, covering login simulation, extracting user profiles, answer likers, followers, avatars, and all answers of a question, and storing the collected data in SQLite, while highlighting challenges like captcha and anti‑scraping limits.
