Skip to content
Tagged articles

web scraping

500 articles · Page 3 of 5
MaGe Linux Operations
MaGe Linux Operations
Jul 3, 2022 · Backend Development

How to Automate 10,000 Video‑Channel Posts with Python and OCR for Massive Traffic

This guide shows how to use Python to scrape high‑quality chat screenshots, apply OCR, generate silent chat videos, batch‑download matching audio from short‑video platforms, and combine them into thousands of unique WeChat Video Channel clips, leveraging volume to outsmart recommendation algorithms and boost traffic.

OCRPythonVideo Generation
0 likes · 11 min read
How to Automate 10,000 Video‑Channel Posts with Python and OCR for Massive Traffic
Python Programming Learning Circle
Python Programming Learning Circle
May 23, 2022 · Backend Development

Simulating Zhihu Login with Python Using urllib and Fiddler

This article demonstrates how to automate Zhihu login on Windows by analyzing network traffic with Fiddler, extracting required parameters, and implementing a Python script that builds HTTP requests using urllib2, handles cookies, captcha retrieval, and logs the results, complete with sample code and execution screenshots.

FiddlerHTTPLogin Automation
0 likes · 8 min read
Simulating Zhihu Login with Python Using urllib and Fiddler
Sohu Tech Products
Sohu Tech Products
May 18, 2022 · Fundamentals

Overview of a Web Page Content Extraction Algorithm and Its Practical Demo

This article introduces a web page content extraction algorithm that automatically structures titles, timestamps, body text, authors, and sources from arbitrary news pages, explains how to use an online demo, compares it with existing solutions, and discusses its broader applications and limitations.

AlgorithmContent ExtractionGNE
0 likes · 8 min read
Overview of a Web Page Content Extraction Algorithm and Its Practical Demo
Programmer DD
Programmer DD
May 15, 2022 · Backend Development

How to Automate Gym Reservations with Selenium: A Google Engineer’s Design Doc

This article explains how a Google engineer designed and implemented an automated system using Python and Selenium to book gym slots two days in advance, detailing problem definition, requirements, architecture, code snippets, and operational workflow for reliable, headless execution on macOS.

PythonSchedulingautomation
0 likes · 11 min read
How to Automate Gym Reservations with Selenium: A Google Engineer’s Design Doc
MaGe Linux Operations
MaGe Linux Operations
May 12, 2022 · Backend Development

7 Fun Python Projects You Can Build in Minutes

This article shares seven practical Python scripts—ranging from a 30‑line Zhihu image scraper and chatbot conversation loop to poetry author classification, lottery number generation, automatic apology letter creation, screen‑capture automation, and GIF assembly—demonstrating how to avoid reinventing the wheel while learning useful automation techniques.

Lottery GeneratorPoetry ClassificationPython
0 likes · 8 min read
7 Fun Python Projects You Can Build in Minutes
Python Programming Learning Circle
Python Programming Learning Circle
May 11, 2022 · Frontend Development

Comprehensive Guide to Configuring Chrome DevTools for Python Web Scraping

This article provides a detailed walkthrough of Chrome DevTools configuration and usage—including global settings, shortcuts, element inspection, network throttling, and code extraction—to help Python developers efficiently collect web data, with step‑by‑step instructions, screenshots, and code snippets.

Chrome DevToolsDebuggingPython
0 likes · 9 min read
Comprehensive Guide to Configuring Chrome DevTools for Python Web Scraping
21CTO
21CTO
Apr 19, 2022 · Information Security

Web Scraping Legalized and Chrome Zero‑Day Patched: Key Tech Updates

Recent developments include a US appellate court affirming that publicly accessible web data can be scraped legally, Google releasing an emergency Chrome 100.0.4896.127 patch for the critical CVE‑2022‑1364 V8 type‑confusion flaw, DB‑Engines’ latest database popularity rankings highlighting Redis’s rise, and Mullvad’s Firefox‑only privacy extension becoming open‑source.

browser securitydatabasesprivacy extension
0 likes · 6 min read
Web Scraping Legalized and Chrome Zero‑Day Patched: Key Tech Updates
Open Source Linux
Open Source Linux
Apr 11, 2022 · Mobile Development

Automate Douyin Video Scraping with Python, mitmproxy, and Appium

This tutorial shows how to combine mitmproxy packet capture and Appium mobile automation in Python to automatically collect and download Douyin video URLs, covering environment setup, code snippets, and practical steps for a fully automated scraper.

AppiumDouyinMobile Automation
0 likes · 8 min read
Automate Douyin Video Scraping with Python, mitmproxy, and Appium
IT Services Circle
IT Services Circle
Mar 26, 2022 · Information Security

Common Reasons Why Your Proxy Fails to Hide Your Web Scraper

The article explains several typical situations—such as not configuring HTTPS proxies, using server IPs, non‑anonymous proxies, polluted IP pools, and lack of HTTP/2 support—that cause websites to easily detect that a request is made through a proxy, even for beginner Python scrapers.

HTTPProxyPython
0 likes · 7 min read
Common Reasons Why Your Proxy Fails to Hide Your Web Scraper
IT Architects Alliance
IT Architects Alliance
Mar 14, 2022 · Backend Development

Design Document: Automated Gym Reservation Bot Using Selenium and Python

This article presents a detailed engineering design document for a Python‑based automation tool that uses Selenium to reserve gym slots during the COVID‑19 pandemic, covering problem description, requirements, overview and detailed design, implementation specifics, and operational workflow.

Gym BookingPythondesign docs
0 likes · 10 min read
Design Document: Automated Gym Reservation Bot Using Selenium and Python
IT Services Circle
IT Services Circle
Mar 7, 2022 · Big Data

Awesome Web Scraping – A Comprehensive Chinese Collection of Web Scraping Resources

This article introduces the renowned "awesome" GitHub repository, highlights its extensive sub‑lists for various domains, focuses on the awesome‑web‑scraping collection, and presents a newly created Chinese version that aggregates Python, JavaScript, Go, and other language‑specific web‑scraping tools and libraries.

Awesome ListGitHubJavaScript
0 likes · 4 min read
Awesome Web Scraping – A Comprehensive Chinese Collection of Web Scraping Resources
MaGe Linux Operations
MaGe Linux Operations
Mar 4, 2022 · Backend Development

How to Build a Local QR‑Code Login Scraper for QQ Music with Python

This tutorial walks through creating a Python‑based local QR‑code login scraper for QQ Music, covering the extraction of dynamic parameters, handling of encrypted cookies, displaying and removing QR images, and ultimately obtaining a usable session for further automation.

Login AutomationPythonQR Code
0 likes · 15 min read
How to Build a Local QR‑Code Login Scraper for QQ Music with Python
Sohu Tech Products
Sohu Tech Products
Mar 2, 2022 · Backend Development

Using requests‑cache to Cache HTTP Requests in Python Web Scraping

This article introduces the requests‑cache library, explains how to install it, demonstrates basic and advanced usage—including session patching, backend selection, expiration policies, request/response filtering, and cache‑control header handling—to efficiently avoid duplicate HTTP requests during Python web scraping.

HTTPPythoncaching
0 likes · 11 min read
Using requests‑cache to Cache HTTP Requests in Python Web Scraping
MaGe Linux Operations
MaGe Linux Operations
Feb 8, 2022 · Backend Development

Automate Qutoutiao Short Video Uploads with Python & Selenium

This tutorial demonstrates how to use Python and Selenium to automatically log in, upload videos and cover images, set titles, descriptions, tags, and publish short videos on the Qutoutiao platform, providing complete source code and step‑by‑step instructions.

PythonQutoutiaoselenium
0 likes · 7 min read
Automate Qutoutiao Short Video Uploads with Python & Selenium
FunTester
FunTester
Feb 8, 2022 · Backend Development

How to Automatically Extract Publication Dates from WeChat Articles with Groovy

The article explains how the author built a Groovy‑based scraper that reads a Markdown list of WeChat links, fetches each article’s HTML, extracts the hidden publication timestamp with a regex, and rewrites the Markdown file to include the dates, using simple HTTP calls and a brief pause to avoid anti‑scraping measures.

GroovyWeChatautomation
0 likes · 6 min read
How to Automatically Extract Publication Dates from WeChat Articles with Groovy
Python Crawling & Data Mining
Python Crawling & Data Mining
Feb 2, 2022 · Backend Development

Bypass SVG Anti‑Scraping and Extract Data with Selenium and requests‑html

This article explains how to scrape data protected by SVG background‑image anti‑scraping by using Selenium to retrieve the SVG URL, parsing the SVG with requests‑html to map background offsets to characters, replacing SVG nodes with text, and finally extracting structured information such as phone numbers and reviews.

SVGdata-extractionrequests-html
0 likes · 11 min read
Bypass SVG Anti‑Scraping and Extract Data with Selenium and requests‑html
MaGe Linux Operations
MaGe Linux Operations
Jan 9, 2022 · Big Data

How to Scrape Maoyan Movie Data and Visualize Trends with Python

This tutorial walks you through collecting movie information from Maoyan using Python web‑scraping, storing the results in CSV, and then applying pandas, matplotlib, and WordCloud to analyze and visualize trends such as release years, genres, regions, durations, and ratings across China and the world.

Movie DataPythonweb scraping
0 likes · 13 min read
How to Scrape Maoyan Movie Data and Visualize Trends with Python
MaGe Linux Operations
MaGe Linux Operations
Dec 30, 2021 · Backend Development

Download Watermark‑Free Douyin Videos with a Simple Python Script

This article explains a streamlined method to extract and download watermark‑free Douyin short videos by inspecting network requests, locating hidden video URLs, and using a concise Python script with the jsonpath library, highlighting its advantages and remaining limitations.

DouyinJSONPathPython
0 likes · 5 min read
Download Watermark‑Free Douyin Videos with a Simple Python Script
Programmer DD
Programmer DD
Dec 28, 2021 · Backend Development

Master Web Scraping with Java: Getting Started with Jsoup

This article introduces Jsoup, an open‑source Java library for extracting and manipulating HTML, explains its key features such as DOM traversal and CSS selectors, and provides a concise code example that fetches Wikipedia headlines, helping developers automate web data collection.

BackendJavadata-extraction
0 likes · 3 min read
Master Web Scraping with Java: Getting Started with Jsoup
Python Programming Learning Circle
Python Programming Learning Circle
Nov 25, 2021 · Big Data

Analyzing and Visualizing Maoyan Movie Reviews for “Chinese Doctors” Using Python

This tutorial demonstrates how to crawl approximately 40,000 Maoyan movie reviews for the film “Chinese Doctors,” preprocess the data, and create visualizations such as rating pie charts, city distribution maps, top‑viewer bar charts, and a word cloud using Python libraries like requests, pyecharts, and wordcloud.

Movie ReviewsPythondata visualization
0 likes · 12 min read
Analyzing and Visualizing Maoyan Movie Reviews for “Chinese Doctors” Using Python
Python Crawling & Data Mining
Python Crawling & Data Mining
Nov 2, 2021 · Backend Development

How to Scrape Global Electricity Capacity Data with Python and Visualize It

Learn step-by-step how to collect worldwide electricity generation data from the IEA using Python's requests and pandas, extract country names, retrieve historical consumption figures, and turn the results into animated visualizations with online tools, while avoiding manual entry and ensuring reproducible analysis.

ElectricityIEAdata visualization
0 likes · 5 min read
How to Scrape Global Electricity Capacity Data with Python and Visualize It
MaGe Linux Operations
MaGe Linux Operations
Oct 23, 2021 · Backend Development

How to Scrape and Analyze Taobao Snack Sales Data with Python

This article walks through a real‑world Python project that uses Selenium to crawl the first ten pages of Taobao snack listings, extracts sales, price and location data, visualizes price distribution and geographic concentration, generates a word‑cloud of top user comments, and lists the top‑selling stores, providing full source code for replication.

PythonTaobaodata analysis
0 likes · 12 min read
How to Scrape and Analyze Taobao Snack Sales Data with Python
Python Crawling & Data Mining
Python Crawling & Data Mining
Oct 11, 2021 · Backend Development

How to Scrape and Analyze 46k Rental Listings with Python: From Crawling to Visual Insights

Learn step‑by‑step how to crawl 46,000+ rental listings from Ziroom using Python, extract house details with regex, clean and transform the data with pandas, and visualize distribution, pricing and location insights through pyecharts, matplotlib and seaborn, revealing rental market patterns in Beijing.

Pyechartsdata cleaningdata visualization
0 likes · 24 min read
How to Scrape and Analyze 46k Rental Listings with Python: From Crawling to Visual Insights
Python Crawling & Data Mining
Python Crawling & Data Mining
Oct 4, 2021 · Backend Development

Automate Shanghai Stock Exchange Report Downloads with a Python Web Scraper

This tutorial shows how to use Python's requests and JSON handling to crawl the Shanghai Stock Exchange website, extract periodic report metadata, construct PDF URLs, and automatically download the files, providing a practical example of backend web‑scraping automation while warning against excessive server load.

PDF downloadSSEweb scraping
0 likes · 28 min read
Automate Shanghai Stock Exchange Report Downloads with a Python Web Scraper
MaGe Linux Operations
MaGe Linux Operations
Sep 18, 2021 · Backend Development

How to Build a Python Crawler to Grab TV Drama Links Automatically

This article explains how to create a Python web crawler that automatically generates URLs for a drama‑download site, filters out invalid pages, extracts ed2k links using requests and regular expressions, saves them to text files, and employs multithreading to speed up processing, while discussing challenges such as duplicate URLs and filename sanitization.

CrawlerMultithreadingregular expressions
0 likes · 7 min read
How to Build a Python Crawler to Grab TV Drama Links Automatically
Python Crawling & Data Mining
Python Crawling & Data Mining
Sep 11, 2021 · Artificial Intelligence

How to Preprocess Captcha Images with OpenCV for Python Scraping

This tutorial explains how to collect captcha images, preprocess them with OpenCV—including grayscale conversion, median blur, binarization, contour detection, and character segmentation—and provides core Python code and visual results for building a reliable Python web‑scraping pipeline.

CAPTCHAOpenCVimage preprocessing
0 likes · 7 min read
How to Preprocess Captcha Images with OpenCV for Python Scraping
Python Crawling & Data Mining
Python Crawling & Data Mining
Aug 26, 2021 · Fundamentals

Transform Raw Taobao Data into Stunning Interactive Charts with Python

This article walks you through cleaning messy Taobao product data using pandas and jieba, then visualizing ingredient and shelf‑life information with interactive Pyecharts charts—including pie, bar, table, funnel, and polar graphs—while showing how to combine multiple charts into a single draggable HTML page.

data visualizationweb scraping
0 likes · 14 min read
Transform Raw Taobao Data into Stunning Interactive Charts with Python
Sohu Tech Products
Sohu Tech Products
Aug 25, 2021 · Backend Development

Scrapy Tutorial: Installation, Project Structure, Basic Usage, and Real‑World Example

This article provides a comprehensive, step‑by‑step guide to the Scrapy web‑crawling framework, covering its core components, installation methods, project layout, spider creation, data extraction techniques, pagination handling, pipeline configuration, and how to run the crawler to collect and store data.

CrawlerPythondata-extraction
0 likes · 13 min read
Scrapy Tutorial: Installation, Project Structure, Basic Usage, and Real‑World Example
MaGe Linux Operations
MaGe Linux Operations
Aug 14, 2021 · Frontend Development

Build a Stunning Python Music Player with PyQt5 – Step‑by‑Step Guide

This tutorial walks you through creating a feature‑rich desktop music player in Python using PyQt5, covering UI layout design, keyword‑based music crawling, multithreaded downloading, playback controls, volume adjustment, and additional functions like random and repeat modes, all illustrated with complete code snippets and screenshots.

Audio PlaybackGUIMultithreading
0 likes · 22 min read
Build a Stunning Python Music Player with PyQt5 – Step‑by‑Step Guide
Python Programming Learning Circle
Python Programming Learning Circle
Jul 14, 2021 · Backend Development

Bypassing Anti‑Scraping Mechanisms: User‑Agent Spoofing and IP Rate Limiting with Python

This article explains how to overcome common anti‑scraping defenses such as identity verification and IP rate limiting by spoofing the User‑Agent header and adding request delays, providing complete Python code examples using requests and BeautifulSoup to scrape Douban's Top 250 movies.

BeautifulSoupIP throttlingUser-Agent
0 likes · 6 min read
Bypassing Anti‑Scraping Mechanisms: User‑Agent Spoofing and IP Rate Limiting with Python
21CTO
21CTO
Jul 12, 2021 · Backend Development

Master Scrapy: From Basics to Advanced Spider Development

This comprehensive guide introduces Scrapy's architecture, explains its core components and data flow, teaches XPath fundamentals, walks through installation, project creation, spider coding, item and pipeline definitions, middleware customization, pagination handling, and essential settings for effective Python web crawling.

CrawlerPythonXPath
0 likes · 14 min read
Master Scrapy: From Basics to Advanced Spider Development
360 Quality & Efficiency
360 Quality & Efficiency
Jul 2, 2021 · Backend Development

Integrating Scrapy with Selenium for Dynamic Web Page Crawling

This guide explains how to combine Scrapy and Selenium to scrape dynamically rendered web pages, covering installation, project setup, middleware configuration, Selenium driver handling, and code examples that demonstrate a complete end‑to‑end crawling workflow.

Dynamic PagesPythonmiddleware
0 likes · 12 min read
Integrating Scrapy with Selenium for Dynamic Web Page Crawling
Python Crawling & Data Mining
Python Crawling & Data Mining
Jun 25, 2021 · Backend Development

How to Scrape NBA Player Stats from Hupu and Auto‑Generate Excel Charts with Python

This guide walks you through building a Python web‑scraper that extracts NBA player information from the Hupu website, cleans and visualizes the data, and automatically creates Excel files with embedded line charts, covering URL navigation, data parsing with requests and BeautifulSoup, and chart generation with xlsxwriter.

BeautifulSoupNBAPython
0 likes · 10 min read
How to Scrape NBA Player Stats from Hupu and Auto‑Generate Excel Charts with Python
MaGe Linux Operations
MaGe Linux Operations
Jun 13, 2021 · Fundamentals

7 Fun Python Projects: Web Scraping, Chatbots, Poetry Classification and More

This article presents seven practical Python scripts—from a concise web scraper for Zhihu images and a chatbot conversation loop to a Naive Bayes poem author classifier, a lottery number generator, an automated essay writer, a screen‑capture tool, and a GIF creator—demonstrating how to avoid reinventing the wheel while exploring diverse automation tasks.

Data GenerationNLPchatbot
0 likes · 8 min read
7 Fun Python Projects: Web Scraping, Chatbots, Poetry Classification and More
MaGe Linux Operations
MaGe Linux Operations
May 7, 2021 · Fundamentals

7 Fun Python Projects to Automate Everyday Tasks (Full Code Included)

This article shares seven practical Python scripts—from web‑scraping Zhihu images and chatting bots to AI poem analysis, lottery generation, automatic apology letters, screen recording, and GIF creation—demonstrating how to avoid reinventing the wheel while learning useful automation techniques.

AICode ExamplesPython
0 likes · 9 min read
7 Fun Python Projects to Automate Everyday Tasks (Full Code Included)