This article introduces a web page content extraction algorithm that automatically structures titles, timestamps, body text, authors, and sources from arbitrary news pages, explains how to use an online demo, compares it with existing solutions, and discusses its broader applications and limitations.
This tutorial shows how to create a Python web crawler that fetches English words, formats them, and automatically emails the list each day using SMTP and a scheduling library, providing a practical solution for daily vocabulary practice.
This article explains how a Google engineer designed and implemented an automated system using Python and Selenium to book gym slots two days in advance, detailing problem definition, requirements, architecture, code snippets, and operational workflow for reliable, headless execution on macOS.
This article shares seven practical Python scripts—ranging from a 30‑line Zhihu image scraper and chatbot conversation loop to poetry author classification, lottery number generation, automatic apology letter creation, screen‑capture automation, and GIF assembly—demonstrating how to avoid reinventing the wheel while learning useful automation techniques.
This article provides a detailed walkthrough of Chrome DevTools configuration and usage—including global settings, shortcuts, element inspection, network throttling, and code extraction—to help Python developers efficiently collect web data, with step‑by‑step instructions, screenshots, and code snippets.
Recent developments include a US appellate court affirming that publicly accessible web data can be scraped legally, Google releasing an emergency Chrome 100.0.4896.127 patch for the critical CVE‑2022‑1364 V8 type‑confusion flaw, DB‑Engines’ latest database popularity rankings highlighting Redis’s rise, and Mullvad’s Firefox‑only privacy extension becoming open‑source.
This article describes how to capture Dingdong's delivery slot API using a mobile packet‑capture tool, set up a Node.js project that polls the API, parse the response, and send iOS notifications through the Bark service, enabling semi‑automatic grocery ordering.
This tutorial shows how to combine mitmproxy packet capture and Appium mobile automation in Python to automatically collect and download Douyin video URLs, covering environment setup, code snippets, and practical steps for a fully automated scraper.
This tutorial demonstrates how to build a Scrapy spider that crawls Zhihu user follow and follower data via Zhihu’s public APIs, handles request headers, parses JSON responses, paginates results, and stores the extracted information into MongoDB using a custom item pipeline.
The article explains several typical situations—such as not configuring HTTPS proxies, using server IPs, non‑anonymous proxies, polluted IP pools, and lack of HTTP/2 support—that cause websites to easily detect that a request is made through a proxy, even for beginner Python scrapers.
This article presents a detailed engineering design document for a Python‑based automation tool that uses Selenium to reserve gym slots during the COVID‑19 pandemic, covering problem description, requirements, overview and detailed design, implementation specifics, and operational workflow.
This article walks through a Python‑based solution for parsing a list of Chinese address strings, extracting province abbreviations, grouping records by province, and optionally using regular expressions or pandas for further analysis, complete with full source code examples.
This tutorial explains the fundamental concepts and parsing principles of XPath, shows how to set up the Python lxml environment, demonstrates instantiating etree objects, details XPath expression syntax, and provides multiple real‑world web‑scraping examples with complete code snippets.
This article introduces the renowned "awesome" GitHub repository, highlights its extensive sub‑lists for various domains, focuses on the awesome‑web‑scraping collection, and presents a newly created Chinese version that aggregates Python, JavaScript, Go, and other language‑specific web‑scraping tools and libraries.
This tutorial walks through creating a Python‑based local QR‑code login scraper for QQ Music, covering the extraction of dynamic parameters, handling of encrypted cookies, displaying and removing QR images, and ultimately obtaining a usable session for further automation.
This article introduces the requests‑cache library, explains how to install it, demonstrates basic and advanced usage—including session patching, backend selection, expiration policies, request/response filtering, and cache‑control header handling—to efficiently avoid duplicate HTTP requests during Python web scraping.
This guide walks you through setting up a Windows environment with PyCharm, mitmproxy, and Appium to capture Douyin video URLs, filter them, and download the videos automatically using Python scripts that drive the mobile app and handle network traffic.
This guide walks you through analyzing Douyin's web pages, bypassing its anti‑scraping checks, constructing the proper API URLs, fetching comment data via Python, storing the results, and visualizing the comments as a dynamic dancing word cloud.
This tutorial shows how to use Python's requests, lxml, and pandas libraries to crawl Weibo's real‑time hot search list, extract ranking, popularity, titles, and detailed comments, and schedule the script to run automatically each hour.
This article walks through creating a Python-based Douyin video downloader that extracts Chrome cookies, parses video URLs, and downloads single videos or entire collections using a simple GUI, complete with code snippets and usage screenshots.
This article introduces the GerapyPlaywright package, explains how to install it, configure Scrapy to use Playwright via middleware and PlaywrightRequest, and provides a complete example spider with code snippets and logging output for JavaScript‑rendered page crawling.
This tutorial demonstrates how to programmatically obtain a QQ Music QR‑code login session using Python by fetching the QR image, extracting and decrypting the required cookies, handling dynamic parameters, and completing the login flow with detailed code examples and explanations.
This guide walks you through extracting text from a CSV obtained via Python web scraping, cleaning it with stop‑words, generating a word‑cloud, performing jieba tokenization and frequency analysis, and finally applying SnowNLP for sentiment scoring, with all code snippets and data links provided.
This tutorial demonstrates how to use Python and Selenium to automatically log in, upload videos and cover images, set titles, descriptions, tags, and publish short videos on the Qutoutiao platform, providing complete source code and step‑by‑step instructions.
The article explains how the author built a Groovy‑based scraper that reads a Markdown list of WeChat links, fetches each article’s HTML, extracts the hidden publication timestamp with a regex, and rewrites the Markdown file to include the dates, using simple HTTP calls and a brief pause to avoid anti‑scraping measures.
This article explains how to scrape data protected by SVG background‑image anti‑scraping by using Selenium to retrieve the SVG URL, parsing the SVG with requests‑html to map background offsets to characters, replacing SVG nodes with text, and finally extracting structured information such as phone numbers and reviews.
This article walks through collecting NetEase Cloud Music hot comments via a Python API, cleaning the data with pandas, and visualizing insights such as comment timing, user activity, word clouds, age distribution, regional maps, and gender ratios using pyecharts.
This tutorial walks you through collecting movie information from Maoyan using Python web‑scraping, storing the results in CSV, and then applying pandas, matplotlib, and WordCloud to analyze and visualize trends such as release years, genres, regions, durations, and ratings across China and the world.
This tutorial walks through using Selenium to log into Alibaba International, scrape supplier phone numbers and related details across multiple pages, save the data to CSV, download product images, and finally embed those images into an Excel workbook for easy reference.
This article explains a streamlined method to extract and download watermark‑free Douyin short videos by inspecting network requests, locating hidden video URLs, and using a concise Python script with the jsonpath library, highlighting its advantages and remaining limitations.
This article introduces Jsoup, an open‑source Java library for extracting and manipulating HTML, explains its key features such as DOM traversal and CSS selectors, and provides a concise code example that fetches Wikipedia headlines, helping developers automate web data collection.
This guide explains how to collect images from Zhihu’s “美女” topic using Python’s Requests and lxml, filter them with Baidu’s AipFace SDK based on face presence, gender, authenticity, and beauty score, and store the qualified pictures locally.
This article demonstrates how to use Python's requests and BeautifulSoup libraries to crawl the Biquw.com novel site, extract chapter lists, download each chapter into text files, handle common anti‑scraping measures, and organize the results in a folder named after the book ID.
This article walks through using Python to scrape Wang Leehom’s divorce‑related Weibo comments, clean the noisy dataset, visualize hourly comment trends, compare with his ex‑wife’s posts, generate word‑clouds and emoji frequency charts, and provides full code and data for reproducible analysis.
This tutorial shows how to crawl the China Meteorological Administration website to obtain today's weather and a six‑day forecast, handle Chinese encoding, extract data with XPath, and save the results into a CSV file using Python's requests and lxml libraries.
This tutorial demonstrates how to use Python on Windows to crawl Maoyan movie rankings, extract details such as title, rating, genre, region, and duration, store them in a CSV file, and then perform comprehensive data cleaning, analysis, and visualization with pandas, matplotlib, and WordCloud.
This tutorial demonstrates how to use Selenium to crawl short comments from a Douban movie page, extract them via XPath, paginate through results, and finally create a visual word cloud using Python's wordcloud library.
This article demonstrates how to use Python's requests and BeautifulSoup to crawl text from a news site, process it with collections, numpy, and jieba for word‑frequency analysis, and then visualize the top terms using pyecharts, providing complete code snippets and explanations.
This article explains a complete Python workflow for solving image‑click captchas by retrieving the base64 image, binarizing it, detecting character contours, generating candidate character images, matching them, and finally returning the click coordinates needed for verification.
This tutorial explains JSON fundamentals, demonstrates Python's json module functions, introduces the JSONPath library for extracting data from JSON documents, and provides a complete example of crawling city information from a website and parsing it with JSONPath.
This article demonstrates how to use Python's Selenium WebDriver to log into Alibaba International, scrape company contact information across multiple pages, save the data to CSV, download product images, and embed them into an Excel file, providing a complete automation workflow.
This guide demonstrates how to use Python on macOS to scrape love quotes, store them, and automatically launch QQ to send a random quote each day, including code snippets and steps to set up an Automator app for startup execution.
This article demonstrates step‑by‑step how to overcome a website’s anti‑scraping defenses using Selenium with stealth options, retrieve CSS‑based price images, reconstruct the digits, and apply Tesseract OCR to accurately extract numeric data, providing complete Python code snippets throughout.
This tutorial demonstrates how to crawl approximately 40,000 Maoyan movie reviews for the film “Chinese Doctors,” preprocess the data, and create visualizations such as rating pie charts, city distribution maps, top‑viewer bar charts, and a word cloud using Python libraries like requests, pyecharts, and wordcloud.
This article explains how font‑based anti‑scraping works, shows how to locate and download custom font files, decode their glyph mappings into a dictionary, and use the mappings to extract real data from a recruitment site with Python, Scrapy and MySQL.
Learn step‑by‑step how to fetch Baidu Index search volume data for any keyword using Python: discover the API endpoints, decrypt the encrypted response with a custom function, and wrap the process into reusable code without relying on UI automation tools.
This article demonstrates a simpler method to obtain water‑marked‑free Douyin video URLs by inspecting network requests, extracting hidden links, and using a Python script with the jsonpath library to download videos directly, highlighting advantages and remaining limitations.
This tutorial demonstrates how to analyze an e‑commerce search page, use Selenium to automate browsing, extract product details across multiple pages, and store the collected data into a MySQL database, providing complete Python code and step‑by‑step explanations.
This guide walks through using Python's requests and lxml libraries to scrape quotes, authors, and tags from quotes.toscrape.com, parse the HTML with XPath, and store the results in a CSV file, covering request headers, element inspection, and data extraction steps.
This article introduces Python 3 methods for retrieving various document types—including TXT, CSV, PDF, and Word files—using urllib, regular expressions, and file‑specific processing steps, providing practical code examples and workflow guidance for building effective web crawlers.
This tutorial explains how to create a Python Selenium script that automatically opens Taobao, logs in via QR code, navigates to the shopping cart, waits for a preset flash‑sale time, selects all items, and completes the checkout process.
This tutorial shows how to build a Python web‑scraper that extracts book titles and authors from a website, formats the data, and automatically sends it via email on a scheduled interval using SMTP and the schedule library.
This article walks through the step-by-step process of extracting individual and batch stock minute-level data from Eastmoney using Python, covering URL analysis, parameter discovery, handling pagination, constructing CSV output, and extending the script to fetch arbitrary dates via cookie manipulation.
Learn step-by-step how to collect worldwide electricity generation data from the IEA using Python's requests and pandas, extract country names, retrieve historical consumption figures, and turn the results into animated visualizations with online tools, while avoiding manual entry and ensuring reproducible analysis.
This tutorial walks you through using Python's built‑in Tkinter library to build a small GUI application that fetches personalized signature images from an online service, downloads them, and displays the result, complete with code examples and UI design tips.
This article demonstrates how to use Python to scrape fund data from Eastmoney, store it in Excel, visualize fund type distribution, and perform detailed time‑series analysis on a specific fund's net‑value, growth rate, and yearly performance.
This tutorial demonstrates how to use Python to scrape Weibo for the most common National Day leave reasons, extract comment data via Ajax requests, store it in CSV files, and create visualizations such as top‑liked users and word‑clouds to reveal popular motifs.
This article walks through a real‑world Python project that uses Selenium to crawl the first ten pages of Taobao snack listings, extracts sales, price and location data, visualizes price distribution and geographic concentration, generates a word‑cloud of top user comments, and lists the top‑selling stores, providing full source code for replication.
Learn step‑by‑step how to crawl 46,000+ rental listings from Ziroom using Python, extract house details with regex, clean and transform the data with pandas, and visualize distribution, pricing and location insights through pyecharts, matplotlib and seaborn, revealing rental market patterns in Beijing.
This tutorial shows how to use Python's requests and JSON handling to crawl the Shanghai Stock Exchange website, extract periodic report metadata, construct PDF URLs, and automatically download the files, providing a practical example of backend web‑scraping automation while warning against excessive server load.
This article walks you through using Scrapy to fetch price data from Beijing Xinfadi's website, covering request analysis, spider creation, item definition, pagination, data extraction, pipeline setup, and exporting results to CSV with full code examples.
This tutorial demonstrates how to use Python's requests and lxml libraries to scrape static web pages, extract image URLs via XPath, convert thumbnail links to full‑size URLs, and download the images to a local folder, providing a complete end‑to‑end example.
This article walks you through using Python's Selenium library to programmatically retrieve Bilibili video series playlists, explains the required XPath selectors, provides a complete, runnable code example, and addresses common issues such as ChromeDriver compatibility.
This article explains how to create a Python web crawler that automatically generates URLs for a drama‑download site, filters out invalid pages, extracts ed2k links using requests and regular expressions, saves them to text files, and employs multithreading to speed up processing, while discussing challenges such as duplicate URLs and filename sanitization.
This article presents a Python‑based web‑scraping project that collects the first ten pages of "小鱼零食" search results on Taobao, analyzes sales, price distribution, geographic store locations, top comments, and visualizes the findings with bar charts and word clouds.
This tutorial walks you through analyzing Tencent's recruitment page, locating the Ajax JSON endpoint, and using Scrapy to create a project, spider, items, pagination, settings, and data export to collect job postings efficiently.
This article explains how to scrape tens of thousands of images from Sogou by analyzing the request URLs, extracting image URLs from JSON responses, and implementing a multithreaded Java downloader that stores the pictures locally.
This tutorial explains how to collect captcha images, preprocess them with OpenCV—including grayscale conversion, median blur, binarization, contour detection, and character segmentation—and provides core Python code and visual results for building a reliable Python web‑scraping pipeline.
This tutorial shows how to use Python to fetch the Weibo hot search list, clean and organize the data with pandas, send the results via QQ email, and schedule the whole process to run automatically each evening.
This tutorial walks you through using Python to collect JD.com product comments, parse the JSON responses, store the data in Excel, and then perform simple visual analysis such as color‑distribution bar charts and a word‑cloud of review content.
This tutorial explains how to install and use the Python jsonpath library to efficiently query and extract values from deeply nested JSON structures, covering basic syntax, indexing, conditional filtering, and result formatting with clear code examples and illustrations.
This article walks you through cleaning messy Taobao product data using pandas and jieba, then visualizing ingredient and shelf‑life information with interactive Pyecharts charts—including pie, bar, table, funnel, and polar graphs—while showing how to combine multiple charts into a single draggable HTML page.
This article provides a comprehensive, step‑by‑step guide to the Scrapy web‑crawling framework, covering its core components, installation methods, project layout, spider creation, data extraction techniques, pagination handling, pipeline configuration, and how to run the crawler to collect and store data.
This tutorial demonstrates how to capture network packets from the Super Schedule Android app, perform a login request with required headers, and repeatedly fetch and parse topic data using Python urllib2 and JSON handling, enabling infinite scrolling of user‑generated topics.
This guide explains how to analyze Sogou's image search XHR request, extract image URLs from the JSON response, and use Java with multithreaded HTTP requests to download thousands of pictures efficiently, including full source code and execution tips.
This tutorial walks through creating a stock sentiment analysis pipeline in Python by scraping news from a financial website, applying Baidu AI's sentiment API to classify headlines, aggregating positive and negative ratios per stock, and visualizing the results with matplotlib.
This article demonstrates how to crawl Bilibili creator data, analyze fan counts, categories, gender and video statistics with Python and pandas, and then create a nickname generator for aspiring million‑follower up‑hosts using both Python and JavaScript.
This tutorial walks through collecting complaint data from the Black Cat platform during the pandemic, cleaning and merging the datasets with pandas, analyzing temporal trends, and generating insightful word clouds using jieba and WordCloud, all illustrated with Python code and visualizations.
This article demonstrates how to scrape attraction and food data for Changsha using Python, process it with pandas, visualize insights with Plotly and Pyecharts, and derive travel recommendations such as top scenic spots, popular eateries, and price trends.
This tutorial walks you through creating a feature‑rich desktop music player in Python using PyQt5, covering UI layout design, keyword‑based music crawling, multithreaded downloading, playback controls, volume adjustment, and additional functions like random and repeat modes, all illustrated with complete code snippets and screenshots.
This article walks you through building a Python web‑scraper that extracts second‑hand housing data from Lianjia, covering target identification, page structure analysis, multi‑level filtering, pagination handling, detailed page parsing, and data storage with practical code examples.
This guide explains how to programmatically log into a website using Python's urllib2 and cookielib modules, covering cookie handling, captcha retrieval, form data construction, request headers, and posting credentials to access protected pages.
This article walks through a Python-driven web‑scraping and data‑analysis workflow that collects, cleans, and visualizes tourism and food data for Changsha, revealing the most popular sights, highly‑rated eateries, and visitor preferences through interactive charts and word clouds.
This tutorial walks you through analyzing a JavaScript‑driven website, discovering the hidden API, configuring request headers with Postman, and writing a Python scraper that extracts book titles, authors, codes, and prices from epubit.com.
This article explains how to overcome common anti‑scraping defenses such as identity verification and IP rate limiting by spoofing the User‑Agent header and adding request delays, providing complete Python code examples using requests and BeautifulSoup to scrape Douban's Top 250 movies.
This comprehensive guide introduces Scrapy's architecture, explains its core components and data flow, teaches XPath fundamentals, walks through installation, project creation, spider coding, item and pipeline definitions, middleware customization, pagination handling, and essential settings for effective Python web crawling.
Learn how to automate a web-based Sudoku game using Python's Selenium for data extraction, implement an efficient backtracking and bitwise algorithm to solve the puzzle, and programmatically fill the solution back into the browser, achieving performance improvements from 17 seconds to just over 3 seconds.
This guide walks you through analyzing Toutiao’s Ajax requests, extracting image URLs, and using Python’s requests library to download street‑photo pictures in bulk, complete with code snippets and tips for pagination and future enhancements.
This guide explains how to combine Scrapy and Selenium to scrape dynamically rendered web pages, covering installation, project setup, middleware configuration, Selenium driver handling, and code examples that demonstrate a complete end‑to‑end crawling workflow.
This tutorial walks you through installing Selenium and ChromeDriver, configuring the environment, and using Python code to automatically navigate Dangdang's bestseller pages, extract book details with pyquery, and save the results into a CSV file for further analysis.
This guide walks you through building a Python web‑scraper that extracts NBA player information from the Hupu website, cleans and visualizes the data, and automatically creates Excel files with embedded line charts, covering URL navigation, data parsing with requests and BeautifulSoup, and chart generation with xlsxwriter.
This article presents seven practical Python scripts—from a concise web scraper for Zhihu images and a chatbot conversation loop to a Naive Bayes poem author classifier, a lottery number generator, an automated essay writer, a screen‑capture tool, and a GIF creator—demonstrating how to avoid reinventing the wheel while exploring diverse automation tasks.
This article demonstrates how websites can detect browsers driven by Selenium or Puppeteer, walks through experiments using bot.sannysoft.com, shows screenshots of detected features in both headed and headless modes, and explains why such tools often fail to evade detection.
This tutorial walks beginners through the complete process of building a Python web scraper to collect internship listings from Shixi.com, covering page analysis, static vs dynamic detection, data location with XPath, pagination handling, and storing results in a CSV file using pandas.
Learn step-by-step how to use Python's Selenium library to automate login, scroll through QQ Space, extract historical posts and download album photos, including driver setup, code snippets for login, shuoshuo retrieval, and photo downloading, enabling you to preserve nostalgic content.
This article shares seven practical Python scripts—from web‑scraping Zhihu images and chatting bots to AI poem analysis, lottery generation, automatic apology letters, screen recording, and GIF creation—demonstrating how to avoid reinventing the wheel while learning useful automation techniques.
This guide explains why Python is essential, presents a step‑by‑step learning roadmap covering beginner basics, backend web development, web crawling, data analysis, and machine learning, and provides curated resources and project links to help learners progress efficiently.
The article presents a comprehensive guide to Python's built‑in functions, standard libraries, and third‑party packages across file I/O, web scraping, databases, data cleaning, statistical analysis, machine learning, visualization, and more, rating each with stars and offering a free e‑book collection for readers.
This tutorial guides readers through analyzing web pages, using Python to write network programs, applying regular expressions, and operating Excel to scrape all listed company stock data for a specified date range, save it by stock code, and optionally store results in databases.
This article provides a comprehensive Python web‑scraping guide covering basic GET/POST requests with urllib2, proxy handling, cookie management, header manipulation to mimic browsers, gzip compression handling, regular‑expression and library parsing, simple captcha strategies, and a multithreaded thread‑pool example.
