This tutorial shows how to build a JD.com search URL, fetch the page with urllib, parse the HTML using BeautifulSoup, and reliably extract each product's name, link, image and price while handling missing image URLs and avoiding regex complexity.
This article explains why Golang’s concurrency and low‑resource footprint make it ideal for web crawling, compares major Go crawling frameworks, and walks through a practical Douban book‑list and comment scraper with complete Colly and Rod code samples, plus a selection matrix and best‑practice summary.
This article walks through a real‑world Python web‑scraping issue where the retrieved map data mismatched the displayed results, explains why the original API call failed, and shows how switching to the correct API and adjusting parameters resolves the inconsistency.
This article introduces Scrapeasy, a Python library that lets you scrape websites, retrieve subpage links, images, PDFs, and videos with just a few lines of code, showing installation, basic usage, and advanced download techniques for various file types.
This tutorial explains how to use Python to scrape product reviews from JD.com via its AJAX comment API, extract fields such as nickname, score, content, and image count, and save the collected data into a CSV file using the requests and csv modules.
This article introduces beginner-friendly Python web‑scraping methods, covering the simplest urllib/requests approach, adding request headers, inspecting network traffic for hidden data, handling dynamically loaded content with Selenium, and provides links to deeper tutorials for each technique.
This tutorial shows how to use Python to scrape a daily article from a website, format the content, and automatically send it via email using SMTP, with full source code and guidance for scheduling the script.
This article explains how to retrieve the names and links of hot songs from NetEase Cloud Music using Python's requests library together with html5lib for HTML parsing, providing full code, handling encoding issues, and comparing it with previous regex, xpath, bs4, and pyquery approaches.
This tutorial explains how to fetch hot song names and links from NetEase Cloud Music using Python's requests, BeautifulSoup (bs4), and a simple HTML cleanup to overcome malformed tags, providing full code and a working example.
This tutorial walks through multiple Python techniques for downloading regular files, web pages, Amazon S3 objects, and other resources, covering basic requests, wget, handling redirects, chunked large‑file downloads, parallel downloads, progress bars, urllib, urllib3, proxy usage, boto3 for S3, and asynchronous downloads with asyncio.
This article explains how to use Python's requests, BeautifulSoup, and Pillow libraries to crawl high‑resolution anime pictures from the Huashi6 website, extract their webp URLs via regex, download them, and convert the images to the more common png format for personal use.
This article walks you through using Python's requests library together with BeautifulSoup to crawl Baidu search results, extract titles and URLs, and save the data into a CSV file, complete with full source code and sample output images.
This tutorial explains how to use Python 3 with Requests, lxml, and Baidu's AipFace SDK to crawl images from Zhihu's "美女" topic, filter them by face detection, gender, authenticity, and beauty score, and store the qualified pictures locally.
This guide demonstrates how to read and write common file types—including TXT, CSV, JSON, XML, Excel, YAML, Parquet, and HDF5—as well as how to connect to SQLite databases and perform web scraping using Python's standard libraries and popular third‑party packages.
This article provides a collection of practical Python code snippets covering file and directory management, data processing, network requests, automation tasks, document handling, image manipulation, system monitoring, visualization, cleaning, logging, and web scraping, all presented with clear explanations and ready-to-use examples.
This tutorial explains how to capture Bilibili video streams with Fiddler, extract the necessary request headers and URLs, and use a concise Python script based on the requests library to download videos, handling header nuances and batch‑fetching video lists from an uploader’s page.
This tutorial walks you through installing the Python requests library, using GET, POST, and PUT methods, handling query parameters, setting custom headers to bypass anti‑scraping measures, and saving both HTML content and images to local files, complete with runnable code examples.
Learn how to use Python to scrape hero equipment data from the King of Glory website, download images via multithreaded requests, and automatically generate comprehensive Markdown and Excel reports, complete with code snippets and troubleshooting tips for a seamless data extraction workflow.
This article walks through a Python‑based solution for extracting Amazon product videos stored as m3u8 streams, explains the format, shows how to retrieve the .ts segments, combine them into an MP4 file, and offers alternative ffmpeg commands for quick downloading.
This article compiles twenty‑three Python web‑scraping projects, each described with its purpose, key features, and a direct GitHub repository link, offering developers a ready‑made toolbox for data collection across platforms such as WeChat, DouBan, Zhihu, Bilibili, and more.
This article demonstrates how to fetch historical lottery data for two games, apply a custom matching rule to identify purchase opportunities, analyze consecutive win patterns over 100 days, and export the aggregated results to an Excel file using Python's requests, xlwt, and basic data‑processing techniques.
This article walks through extracting JSON data from a web page using Python, showcasing multiple code examples with requests, regex, BeautifulSoup, and json handling, and explains how to simplify storage and avoid common parsing pitfalls for reliable web scraping.
This article outlines essential Python web‑crawling techniques—including basic GET/POST requests, proxy usage, cookie management, header spoofing, captcha handling, gzip compression, and multithreaded fetching—to help developers build efficient and robust crawlers.
This article outlines essential Python web‑scraping techniques, covering basic GET/POST requests, proxy usage, cookie handling, header manipulation to mimic browsers, simple captcha solutions, gzip compression handling, and multithreaded crawling with a thread‑pool template, providing practical code examples for each step.
Learn how to overcome anti‑scraping defenses by using Python's requests_html library to fetch and parse dynamic web pages, with a complete code example that extracts company names from a target site, plus tips on handling cookies, headers, and Unicode decoding.
This article describes a Python Flask‑based system that crawls tourism site reviews, processes the data with pandas, stores it in MySQL, and visualizes sentiment analysis, keyword frequencies, and rating trends through interactive ECharts charts on a responsive web dashboard.
This article presents ten practical Python scripts—including data analysis with Pandas, web scraping with BeautifulSoup, file renaming, image resizing, PDF generation, email sending, data backup, password generation, a simple HTTP server, and SQLite backup/restore—to help automate common programming tasks.
This article presents seventeen versatile Python scripts that automate tasks such as file management, web scraping, email handling, database interaction, system monitoring, and image processing, providing clear code examples and explanations to help developers and analysts streamline workflows and boost productivity.
This article provides a comprehensive Python tutorial for scraping second‑hand housing community data from Anjuke, covering city selection, URL collection, HTML parsing with lxml, data cleaning, CSV export, and full‑city crawling strategies, complete with runnable code examples.
This article explains a simplified Python method for extracting and downloading watermark‑free Douyin videos by inspecting network requests, using the jsonpath library, and automating the download process with concise code, while also discussing its advantages and current limitations.
This article walks through a common Python web‑scraping issue where data from multiple pages is overwritten, explains the root cause, and provides a complete, runnable script using requests, parsel, and openpyxl to correctly collect and store each page's information.
This article introduces ten practical Python techniques—including requests, Selenium, Scrapy, Crawley, PySpider, aiohttp, asks, vibora, Pyppeteer, and Fiddler‑based reverse engineering—to efficiently collect e‑commerce and app data while addressing common challenges such as IP blocking, captchas, and authentication.
This article walks through a Python web‑scraping solution using requests and pandas to fetch Eastmoney stock data, explains the limitations of AI‑generated code, and provides a complete, runnable script with sample output.
This article walks through a Python web‑scraping problem, explains why you‑get downloads may fail, and provides a clear, step‑by‑step solution—including firewall checks and network tweaks—to get the video files successfully.
This article presents nine useful Python automation scenarios—including file management, web scraping, text processing, email handling, Excel manipulation, database interaction, system task scheduling, and image editing—each accompanied by clear script examples and explanations to help readers streamline repetitive workflows efficiently.
This article introduces the Scrapeasy Python library, explains how to install it with a single pip command, and demonstrates step‑by‑step code examples for initializing websites, extracting links, images, videos, and other files, highlighting its ease of use for fast web data extraction.
This article explains how to resolve a Python web‑scraping issue for Douyin hot videos by keeping the cookie unchanged and updating the request headers, particularly the Referer, allowing the script to successfully fetch and display video information.
This article presents a concise guide to Python web scraping, covering eight practical techniques—including basic GET/POST requests, proxy usage, cookie management, header spoofing, page parsing, captcha handling, gzip compression, and multithreaded crawling—each illustrated with clear code examples.
This article presents a practical Python web crawler that retrieves Douban movie data, offers ranking and keyword search, displays details via a Tkinter GUI, lists free streaming sources, and discusses configuration steps, features, known issues, and core code snippets.
This article demonstrates how to use Puppeteer and the Canvas API to automate login on Juejin, extract the slider captcha image, apply grayscale and binarization processing to locate the gap, calculate the required drag distance, and simulate human‑like mouse movements with easing functions for successful verification.
This article explains how to automate Google SEO keyword rank checking by writing a Python script that sends HTTP requests, parses the search results with BeautifulSoup, iterates through results to find a target domain, and handles errors, providing a free alternative to costly SEO tools.
This article walks through a step‑by‑step Python solution for scraping Shanghai Gold Exchange transaction data, using requests_html to collect URLs, pandas to parse HTML tables, and finally exporting the aggregated results to Excel, while also offering practical tips for sharing code and debugging.
This article explains how to collect images from Zhihu's "beauty" topic using Python's Requests and lxml libraries, filter them with Baidu AI's AipFace face detection service, and store the qualified pictures locally, detailing the required environment, logic, and preparation steps.
This article walks through a Python web‑crawler error reported in a community chat, shows the original code snippet, explains how downgrading the Python version resolved the issue, and offers practical tips for sharing reproducible examples when seeking help.
This article presents a curated list of Python web‑scraping utilities—including file download assistants, novel and video grabbers, proxy pool builders, and various automation scripts—along with installation commands, usage examples, source links, and brief operational explanations for each tool.
This article presents a collection of seven practical Python scripts—including a Zhihu image scraper, two interacting chatbots, a Naive Bayes poetry author classifier, a 35‑choose‑7 lottery generator, an automatic apology writer, a screen‑capture tool, and a GIF maker—each demonstrated with complete, runnable code.
This article demonstrates how to use Python's requests library to call the YY live video API, parse the returned JSON for video URLs, download each video file, and store them locally, while also providing a complete reusable script for repeated batch downloads.
This article walks through solving Python web‑scraping captcha challenges by using Selenium to capture the image, applying OCR for recognition, and offering alternative request‑based methods, while also addressing common driver version mismatches.
This tutorial explains how to scrape a news article with Python, segment Chinese text, count word frequencies, and visualize the top ten words using a word cloud and a pie chart, providing complete code and sample results.
This article introduces ten useful Python automation scripts—including file backup, batch renaming, email sending, text conversion, image processing, timed monitoring, web scraping, folder organization, note compilation, and screen capture—to help readers automate repetitive tasks and boost productivity.
This article showcases Python's concise syntax and powerful libraries by comparing a simple web request with JavaScript and providing five practical automation scripts—including a URL shortener, fake data generator, YouTube downloader, NATO alphabet encoder, and Selenium-based social‑media login—demonstrating why Python is ideal for repetitive tasks.
This article walks through a Python‑based Baidu web‑scraping solution, explaining the problem, showing the debugging process with screenshots, and providing a complete, runnable script that extracts keywords, titles, URLs, descriptions, and site names into a CSV file.
This article walks through a real‑world Python web‑scraping problem raised in a community, presents a clear solution with parameter adjustments, shows before‑and‑after results, and highlights common pitfalls to help developers reliably extract data.
This article explains three practical methods—using response.content, apparent_encoding, and custom encode/decode—to resolve Chinese character garbling in Python web crawlers, includes code snippets and screenshots, and offers concise guidance for developers facing this common issue.
This article explains how to use a Python program to parse Tencent video URLs, retrieve the cached TS segments via HTTP requests, download them concurrently with multiprocessing, and finally merge the segments into a playable MP4 file, including environment setup and full source code.
This article walks through a real‑world Python web‑scraping problem, demonstrating how to extract clean table data using pandas.read_html and an alternative XPath approach, complete with code snippets, output screenshots, and practical tips for robust data collection.
This article presents a series of Python automation examples—including HTTP requests, URL shortening, fake data creation, YouTube video downloading, NATO‑style message encryption, and Selenium‑driven social‑media login—showcasing concise code snippets and explanations that highlight Python's simplicity and versatility for repetitive tasks.
This article walks through a Python web‑scraping solution that extracts GDP data from a website using requests, lxml, and pandas, then stores the results in a CSV file, providing complete code and step‑by‑step explanation for readers.
This article introduces Scrapeasy, a Python library that enables one‑line web scraping, extraction of images, videos, PDFs and other files, and demonstrates installation, basic usage, subpage link retrieval, media downloading, and advanced link handling with clear code examples.
This article provides a detailed tutorial on Python's urllib library, covering its main modules (request, error, parse, robotparser), key functions and classes, code examples for URL fetching, parsing, encoding, and handling robots.txt, making it a practical resource for backend developers and web scrapers.
This article presents ten practical Python automation scripts—including HTML parsing, QR code scanning, screenshot capture, PDF manipulation, mobile device control, CPU/GPU temperature monitoring, Instagram uploading, and video watermarking—to help developers eliminate repetitive tasks and streamline everyday workflows.
This guide explains how unemployed developers can use Python, especially web‑scraping techniques, to secure freelance gigs by leveraging various platforms, community groups, and effective order‑taking strategies while warning about common pitfalls and anti‑scraping challenges.
This tutorial demonstrates how to use Python's requests and BeautifulSoup libraries to scrape the titles and release years of the 250 movies listed on Douban, clean the extracted data, output it for Excel, and then create a pivot table and chart to visualize the yearly distribution of top films.
This article demonstrates how to use Python's requests and lxml libraries to scrape web pages and apply regular expressions for extracting both integer and floating‑point numbers, providing clear code examples and step‑by‑step explanations.
This article presents a comprehensive guide to Python web scraping, covering basic GET and POST requests with urllib2, using proxy IPs, managing cookies, disguising as a browser via custom headers, handling gzip-compressed responses, and accelerating crawls with a simple multithreaded worker pool.
This article explains how to use advanced XPath expressions in Python web scraping to exclude unwanted HTML elements, providing step‑by‑step code examples and practical tips for filtering tags based on style attributes and combining logical operators.
This article demonstrates how to use Python's requests library and BeautifulSoup to inspect webpage source, set request headers, fetch weather page HTML, parse it with CSS selectors, extract daytime and nighttime temperatures, and extend the script to handle multiple cities, providing complete code examples.
This tutorial explains how to use Python to automatically extract image URLs from a webpage via developer tools and regular expressions, then download all images in bulk with a simple script that handles sessions, headers, and file saving.
This article demonstrates how to use Python, urllib, BeautifulSoup, regular expressions, and multithreading to crawl the DouTuBa website, extract the URLs of emoji images, and download over a hundred thousand pictures automatically.
This article walks through a step‑by‑step Python web‑scraping tutorial that fetches financial data of over 3000 listed companies from a public website, parses the tables with pandas, enhances the script with error handling, URL flexibility, MySQL storage, and multiprocessing to speed up the crawl.
This article demonstrates two Python approaches—using high-level requests and low-level HTTPConnection—to fetch intraday stock data from Eastmoney, providing complete code examples, URL discovery steps, and practical tips for data‑mining enthusiasts.
This article demonstrates how to scrape popular song titles and links from NetEase Cloud Music using Python's html5lib parser, covering the challenges of XPath and pyquery selectors, code implementation, encoding fixes, and a complete working example.
This article introduces fifteen widely used Python libraries covering data collection, cleaning, visualization, machine‑learning, and web development, providing a comprehensive toolkit for data scientists and analysts to build end‑to‑end data pipelines.
This article demonstrates a simpler method for extracting and downloading watermark‑free Douyin (TikTok) videos by inspecting network requests, locating the hidden video URLs, and using a Python script with the jsonpath library to automate the retrieval process.
This tutorial demonstrates a complete Python script that uses requests and lxml's XPath to fetch Baidu search result titles and URLs, saves them to a CSV file, and includes sample output screenshots for verification.
This tutorial demonstrates how to use Python's requests library together with BeautifulSoup to crawl Baidu search results, extract titles and URLs, and save the data into a CSV file, providing complete code and step‑by‑step explanations.
This article presents a Python solution that parses Tencent video URLs, retrieves encrypted TS video segments through a third‑party VIP service, downloads them concurrently, and merges the segments into a playable MP4 file, detailing the required environment, workflow, and complete source code.
This tutorial demonstrates how to fetch, clean, and organize King of Glory item data via a public JSON API using Python requests and pandas, then download item images concurrently, and finally generate comprehensive Markdown, Excel, and other documentation formats for easy reference.
This article demonstrates how to create a simple desktop translator application in Python using Tkinter for the graphical interface and web‑scraping techniques to call Youdao's translation service, providing complete source code and step‑by‑step explanations.
This article demonstrates how to use Python's requests and BeautifulSoup libraries to scrape current and 14‑day weather data from China Weather Network, store it in CSV files, and then apply pandas, numpy, and matplotlib for comprehensive visual analysis of temperature, humidity, air quality, wind direction, and climate distribution.
This article explains how to use the Python requests_cache library to mimic browser caching behavior in web‑scraping tasks, reducing request latency by skipping delays when cached responses are available, and shows various configuration options, back‑ends, and practical code examples.
This article walks through using the requests library’s stream=True parameter in Python to efficiently filter valid URLs during web scraping, presenting two practical methods, code examples, performance insights, and a clear explanation of how stream handling works.
This article walks through a Python fan's question about storing JSON from web pages during crawling, presents four step‑by‑step code solutions—including regex, BeautifulSoup, and json.loads—and offers practical tips for clean extraction and reliable scraping.
This tutorial demonstrates two concise Python projects—using pyautogui to keep the computer awake by moving the mouse and employing pandas to scrape tabular data from a website—each implemented with roughly five lines of code and explained step by step.
This tutorial demonstrates how to use Python libraries such as requests and Pillow to scrape individual Chinese characters, resize them, compose them onto a background image, and output a customized "Fu" couplet image, complete with full source code and usage instructions.
This tutorial demonstrates how to collect the top 2000 entries of the 2021 China Hurun Rich List using Python's requests library, process the data with pandas, and create insightful visualizations in Tableau, covering data extraction, cleaning, and analysis steps.
This tutorial shows how to build a Python script that scrapes daily recipe articles from a website, formats the content, and automatically sends it via email using SMTP, providing a practical example of web scraping and email automation.
This tutorial walks you through building a Python scraper that extracts video titles and URLs from Pear Video, then downloads the videos to your computer using single‑threaded requests, a thread‑pool, and asynchronous asyncio techniques, complete with full code examples.
This tutorial walks through a complete Python workflow that downloads a B‑site video, extracts frames with OpenCV, uses Baidu AI for human segmentation, crawls danmu comments, creates a masked word‑cloud animation, and finally merges the clips with audio into a polished video.
This article explains how to automate web article extraction by first using BeautifulSoup and urllib to crawl search results and retrieve page links, then demonstrates a much simpler approach with the Python newspaper library to directly download and parse article text, including full installation and code examples.
This article demonstrates how to collect historical Beijing weather data using Python web‑scraping, clean and transform it with pandas, visualize overall and monthly patterns, and extract insights such as the first winter snowfall each year and annual snow‑day counts.
This article walks through decoding custom font anti‑scraping on a Chinese group‑buying site using Python, covering page fetching, CSS font URL extraction, .woff download, glyph mapping with fontTools, OCR via ddddocr, DOM manipulation with BeautifulSoup, and exporting the cleaned data to Excel.
This article walks through a real‑world Python web‑scraping case, explaining the problem, revealing hidden request URLs, showing the complete code, and demonstrating how to retrieve and verify data from Baidu's Aiqicha platform.
This tutorial explains how to use Python 3 with Requests, lxml, and Baidu's AipFace SDK to crawl images from Zhihu's "beauty" topic, filter them by face detection and gender criteria, and store the qualified pictures locally.
This report describes a Python‑based web application built with Flask that scrapes hot‑search data from Weibo, Baidu and Zhihu, processes it using jieba and other libraries, and visualizes the results together with domestic COVID‑19 statistics using ECharts on a responsive front‑end page.
This tutorial explains how to scrape daily inspirational or love quotes from a Chinese website with Python, display them using EasyGUI, and package the script into a standalone executable using PyInstaller, including troubleshooting tips for common packaging errors.
This article demonstrates how to employ Python, Selenium, and web‑scraping techniques to log into QQ Space, extract authentication cookies, compute the required g_tk parameter, retrieve the friend list via QQ's API, and automatically detect and blacklist users who have blocked you.
This guide shows how to use Python tools like you‑get and a custom BililLocal player to download Bilibili videos, handle download issues, retrieve danmu subtitles, and play the content locally, providing step‑by‑step instructions and useful resources.
This tutorial shows how to use Python to crawl Douban's Top250 movie list, handle anti‑scraping measures, extract detailed fields, store the data in Excel, and perform data cleaning, statistical analysis, and visualizations such as year distribution, rating trends, and genre word clouds.
This article introduces a Python-based project that demonstrates how to log into and scrape data from 18 major websites—including Facebook, Twitter, Zhihu, and Bilibili—using methods such as Selenium, direct HTTP requests, and cookie management, providing code examples and future improvement plans.
