This article introduces eleven powerful Python libraries for web scraping—including Requests, BeautifulSoup, Scrapy, Selenium, PyQuery, Lxml, Pandas, Pyppeteer, aiohttp, Faker, and ProxyPool—explains their key features, provides ready‑to‑run code snippets, and demonstrates a real‑world news‑site crawling case study.
This article demonstrates how to use Python's requests and BeautifulSoup libraries to scrape current and 14‑day weather data from China Weather Net, store the results in CSV files, and visualize temperature, humidity, air quality, and wind patterns using Matplotlib, NumPy, and Pandas.
This article walks through a real‑world Python web‑scraping issue where Chinese characters appear as garbled text, explains why the default encoding fails, and shows how setting the response’s apparent encoding resolves the problem, complete with sample code and practical tips for posting questions.
This article explains the PHP get_headers() function, its syntax, parameters, and usage examples, demonstrates how to retrieve and display HTTP response headers from a URL, and discusses common application scenarios such as checking remote file existence, obtaining file metadata, and supporting web crawling or monitoring.
This article presents a Python‑based project that continuously crawls Weibo hot‑search data, stores it with timestamps, and visualizes trends through dynamic bar, line, and word‑cloud charts using libraries such as BeautifulSoup, pandas, schedule, pyecharts, and jieba.
This article provides a concise Python cheat sheet covering common file and directory operations, CSV/JSON data handling, web request and crawling techniques, automation tasks such as scheduling and email sending, and various string manipulation utilities, each illustrated with ready‑to‑run code snippets.
Learn step‑by‑step how to extract text from a CSV generated by a Python web crawler, clean it with stop‑words, create a word‑cloud visualization, compute word frequencies, and perform sentiment analysis using jieba and SnowNLP, with all code snippets provided.
This article walks through a Python web‑scraping project that fetches skin images, names, and prices from Honor of Kings, covering project background, required tools, site analysis, code implementation, data processing, and legal considerations.
This article explains how to build a Python script that searches a comic website, extracts chapter links and image URLs using requests and BeautifulSoup, and downloads the images into organized folders with multithreaded support, while outlining required modules and potential improvements.
This guide walks you through extracting novel chapters from the Biquw.com website with Python, explaining how to obtain the book ID, retrieve the chapter list, download each chapter using requests and BeautifulSoup, handle common anti‑scraping measures, and organize the downloaded files.
This tutorial walks through creating a Coze bot that fetches a Juejin author's information and articles via web scraping, processes the data to generate a concise author profile, identifies expertise domains, ranks hot articles, and finally publishes the bot for interactive use.
This article introduces seven powerful Python libraries—Pandas, Selenium, Flask, Scrapy, Requests, Faker, and Pillow—explaining their core features, typical use cases, and providing ready‑to‑run code snippets to help developers boost productivity and automate routine tasks.
This article presents 17 versatile Python scripts that automate common tasks such as file management, web scraping, email handling, Excel processing, database interaction, system administration, and more, offering code examples and explanations to help developers and analysts improve productivity.
This article explains a Python web‑scraping issue where Selenium sometimes returns only previously scraped rows after page navigation, and provides a step‑by‑step solution using requests, pagination parameters, regex extraction of IDs, and PDF download handling to reliably collect data.
This article walks through a Python community member's problem of turning raw HTTP header strings and cookies into clean dictionary objects, explains why newline characters cause issues, and provides step‑by‑step code solutions using string replacement, JSON conversion, and built‑in libraries.
This article demonstrates how to use Python to scrape Weibo comments about the TV series 'The Legend of the Year', generate word clouds, compute character frequencies and co‑occurrences, and visualize the resulting character relationship network with Gephi.
This tutorial demonstrates how to install Selenium, log into QQ Space, and programmatically retrieve both status updates and album photos using Python's Selenium and BeautifulSoup libraries, including detailed code examples for login, scrolling, and image downloading.
This article presents six practical Python techniques—including exception handling, list comprehensions, regular expressions, web scraping with BeautifulSoup, lambda functions, and simple data visualizations using matplotlib—to help developers write cleaner, more efficient code and enhance their programming workflow.
This article explains how the PHP cURL library, especially the curl_exec() function, can be used to initialize a session, set options, execute HTTP requests, handle errors, and retrieve remote page content efficiently.
This article walks through a Python web‑scraping solution for downloading individual files from a site, presenting the problem, multiple approaches, and a complete, ready‑to‑run script, while also offering tips for handling large data and avoiding IP blocks.
The article showcases a variety of Python scripts that automate tasks such as subtitle and video downloading, IMDb data extraction, email notifications, PNR status checks, WhatsApp avatar monitoring, face recognition for photo sorting, and a custom DC++ sharing tool, providing code examples and usage details for each utility.
This article reviews the ten most essential Python libraries for data science, covering data acquisition, analysis, machine learning, and visualization, and provides concise code examples to help beginners quickly start using tools like Beautiful Soup, NumPy, pandas, scikit‑learn, TensorFlow, Keras, Matplotlib, and seaborn.
This article explains how to obtain cookies from an initial HTTP response using Python's requests library, reuse them in subsequent requests, and handle related security considerations such as CSRF tokens and robots.txt compliance, providing a complete code example and step‑by‑step guidance.
This article explains how combining AI with Node.js crawling through the x‑crawl library makes web data collection more intelligent, flexible, and resilient to website changes, and provides detailed feature descriptions, code examples, and practical usage scenarios.
This article walks through solving a Pandas data‑analysis problem raised in a Python community, showcasing multiple Python techniques—including string manipulation, zero‑width spaces, and slicing—to clean and transform data, complete with code snippets and result screenshots.
This article demonstrates how to use Python's requests, lxml, and pandas libraries to crawl GDP data from a website, parse the HTML tables, and efficiently write the extracted rankings, regions, GDP values, and years into a CSV file, providing a complete, runnable example for web scraping beginners.
This article presents a collection of useful Python scripts—including file sorting, empty‑folder removal, batch renaming, web‑scraping, bulk image downloading, form automation, and various text‑processing utilities—each accompanied by clear explanations and ready‑to‑run code examples.
This article explains how to overcome JavaScript-rendered web pages when scraping with Python, comparing tools like Selenium, Playwright, and the DrissionPage library, and provides a concrete code example that fetches and prints page HTML, along with practical tips for handling JSON data sources.
This article explains why Chinese characters often appear garbled when using Python web crawlers and presents three practical solutions—switching to response.content, manually setting the response encoding, and applying a universal encode‑decode trick—to reliably decode GBK‑encoded text.
This article demonstrates how to use Jsoup to fetch and parse HTML content from a web page, extract specific table data via CSS selectors, map the data to Java objects, and efficiently write the results to an Excel file with EasyExcel, highlighting memory advantages over POI.
This article explains why a Python web crawler often fails to retrieve JavaScript‑rendered page content, outlines common causes such as dynamic loading, anti‑scraping measures, and server‑side rendering, and offers practical techniques to capture the fully rendered HTML.
This article presents nine practical Python automation scenarios—including file management, web scraping, text processing, email handling, Excel manipulation, database interaction, social media posting, system task scheduling, and image editing—each accompanied by clear script examples and explanations to help readers streamline repetitive tasks efficiently.
This guide explains how to install the Cloudmersive PHP SDK with Composer, obtain a free API key, construct a JSON request, and run a PHP script that calls the Cloudmersive Document Convert API to retrieve the full textual content of any web page URL.
This tutorial walks you through installing the Python requests library on Windows and Linux, demonstrates how to perform GET, POST, PUT requests, pass parameters, bypass anti‑scraping measures, and save fetched HTML or images to local files with clear, runnable code examples.
Learn the core concepts of using proxies in Python web scraping, including what proxies are, common types like anonymous and high‑anonymity, how they protect your crawler, practical implementation with the requests library, and an overview of building a proxy pool for scalable data extraction.
This article walks through a five‑minute Python web‑scraping tutorial that fetches recipe pages, parses HTML with urllib and BeautifulSoup, extracts titles and images, and wraps the workflow into a simple console application for random meal selection.
This tutorial explains how to create a simple Flask web site and RESTful API, then demonstrates retrieving the table data using Excel's web import feature and Python's requests, BeautifulSoup, and pandas libraries, while comparing the two approaches.
This article walks through creating a Coze chatbot that recommends weekend attractions and events by designing prompt‑based workflows, leveraging official plugins, scraping city‑level data, constructing a knowledge base, handling pagination with code nodes, and finally deploying the bot for real‑time user queries.
This article reviews the key concepts from the Bad Data Handbook, covering noise identification, data validation, human readability, web data restructuring, special domain challenges, and data quality analysis, while also presenting practical data visualization techniques, popular analysis tools, Python web‑scraping libraries, and a basic NLP workflow with code examples.
This article introduces the popular 12306 ticket‑grabbing assistant, explains its history and features, provides step‑by‑step installation and configuration instructions—including required dependencies, proxy settings, and email notifications—and shows how to run the script to automatically secure train tickets during the Spring Festival travel rush.
This article walks through a common Python web‑scraping text‑encoding problem, provides a complete code example using requests and BeautifulSoup, shows screenshots of the solution, and offers practical tips for asking similar technical questions in the community.
This article presents 17 practical Python scripts covering file management, web scraping, email handling, Excel processing, database interaction, system tasks, image editing, and more, each with code examples and explanations, enabling developers and analysts to automate repetitive workflows and boost productivity across diverse domains.
This article demonstrates how to use LangChain, LLMs, and SerpAPI to create a Weibo influencer‑search tool that extracts UID numbers, scrapes profile data, filters Chinese content, and prepares the information for automated marketing outreach.
This article explains how to set up a web crawler using PHP and Selenium, covering installation of Selenium and its PHP bindings via Composer, configuring a Chrome WebDriver, simulating user actions to fetch news links, extracting titles and content, and storing results, with tips for further optimization.
This article presents a collection of Python code examples demonstrating how to scrape, process, visualize, and analyze data from news sites, social media, stock markets, e‑commerce, web traffic, text, images, and more, covering tasks such as clustering, time‑series forecasting, and sentiment analysis.
This article introduces Scrapeasy, a Python library that enables one‑line web scraping, media extraction, and file downloading, and provides step‑by‑step code examples for installing the package, initializing websites, retrieving links, images, videos, and PDFs, making data collection fast and easy.
Explore 17 practical Python scripts that automate tasks ranging from file management and web scraping to email handling, database interaction, system monitoring, and cloud services, enabling developers and analysts to boost productivity, reduce errors, and streamline workflows across diverse domains.
This article walks through a Python web‑scraping tutorial that extracts ETF codes and names from Eastmoney, detailing the problem, step‑by‑step code using requests, pandas, and handling pagination, and concludes with a brief promotion of a data‑collection service.
This article walks through retrieving ETF fund codes and names from Eastmoney using Python's requests and pandas, explains constructing the correct URLs, handling pagination, cleaning the JSON response, and provides complete sample scripts, while also highlighting a simpler solution and recommending a data‑collection platform.
This article explains how to write a Python crawler that uses the WeChat public platform's open API to obtain an access token, retrieve the list of articles from a specified public account, and extract each article's title, summary, and link for further analysis.
This article introduces Requestium, a Python library that merges Selenium and Requests, and provides step‑by‑step examples covering basic session setup, installation, dynamic content handling, user interaction simulation, cookie and header management, asynchronous fetching, iframe switching, alert handling, and screenshot or video recording.
This guide shows how to use LangChain with OpenAI’s API, Puppeteer, and vector stores to automatically read local or web‑based API documentation, split and retrieve relevant text, and prompt an LLM to generate ready‑to‑use TypeScript front‑end code, highlighting setup, prompt design, and example outputs.
Using Tencent's Hunyuan model, the tutorial walks through a Python workflow that scrapes a student‑score table from a web page, saves it as CSV and Excel, cleans missing values, computes total and average scores, and visualizes their distributions with matplotlib, illustrating how LLMs can accelerate data‑analysis coding while still needing human verification.
This tutorial explains how to use Python with requests and BeautifulSoup to locate image URLs in a webpage's HTML, extract them via regular expressions, and programmatically download all images to a local folder in an automated batch process.
This article demonstrates how to scrape a news webpage with Python, extract and segment its Chinese text using jieba, count word frequencies, and visualize the top ten terms as a word cloud and a pie chart with pyecharts.
This guide explains how to capture Douyin dynamic wallpaper video URLs with mitmproxy, extract MP4 links from the generated content.data file, deduplicate them, and download the videos using a Python script with requests and tqdm.
This tutorial shows how, with about 50 lines of Python using Playwright and pyperclip, you can automate a persistent Firefox session to log into ChatGPT Plus, navigate to your private GPT URL, send prompts and retrieve responses via the clipboard, all without incurring extra API fees.
This article explains several Python techniques for extracting browser cookies—including Selenium automation, direct HTTP requests, driver‑based retrieval, and HAR file parsing—providing clear code examples and guidance on when to use each method.
This article explains what Python web crawlers are, examines the legal and ethical issues surrounding their use, offers practical guidelines for lawful scraping, and provides a comprehensive learning roadmap with tools, techniques, and real‑world scenarios.
This article demonstrates how to use Python's requests and BeautifulSoup libraries to scrape current and 14‑day weather data from China Weather, store the results in CSV files, and perform comprehensive visual analysis with pandas, numpy, and matplotlib, including temperature, humidity, AQI, wind radar, and correlation plots.
This article compiles a variety of Python web‑scraping utilities—including file download assistants, video downloaders, proxy IP pool builders, captcha bypass scripts, and game data extractors—providing descriptions, installation commands, usage examples, download links, and reference links for each tool.
The article examines how Python Requests, Scrapy, and Go’s charset package detect page encodings, reveals why they often mis‑decode Chinese GB‑series pages, and proposes a unified strategy—prefer header charset, then HTML meta, finally a reliable heuristic—to eliminate garbled text in web scraping.
This tutorial explains how to set up the PHP environment, install the Goutte library, and use it to fetch page content, extract hyperlinks, and submit forms, providing complete code examples for building a functional web crawler.
This article walks through discovering Bilibili's comment API, programmatically fetching paginated JSON data, converting it into Java POJOs, storing and sorting the comments, and finally feeding the top entries to ChatGPT for automated sentiment and content analysis.
This tutorial explains process and thread fundamentals, compares single‑threaded and multithreaded Python crawlers, provides complete code examples for both approaches, and demonstrates how converting a single‑threaded scraper to multithreading can significantly reduce execution time when handling large data volumes.
Using Python's Selenium library, this tutorial demonstrates how to scrape real-time box office data from Maoyan's regular page, extract movie names, total and incremental earnings, process the data with pandas, export to Excel, and create visual analyses of top‑10 films' revenues and market shares.
This article explains how to enable the PHP cURL extension, create an array of target URLs, and use curl_multi functions with example code to perform concurrent HTTP requests and collect their responses efficiently.
This article walks through building a Python web scraper that automatically logs in, fetches daily free reports from a registration‑required site, saves each as a PDF with appropriate filenames, and includes complete source code with detailed explanations for handling retries, cookies, and file management.
This article introduces ten widely used Python libraries—NumPy, Pandas, Matplotlib, Scikit‑learn, Requests, Beautiful Soup, Selenium, Pygame, Flask, and PyTorch—each accompanied by a concise real‑world project and complete code examples to help readers understand and apply them effectively.
This article explains how to use Selenium's window handles and switch_to.window method to perform repeated debugging within the same browser instance, providing a full Python example that logs in, switches windows, and returns to the original session without closing the browser.
This tutorial explains how to install Google Chrome and ChromeDriver on CentOS, use PHP‑WebDriver to scrape dynamic web pages, and introduces XPath syntax for locating elements, providing step‑by‑step commands, code samples, and execution results.
Scrapy is a powerful Python-based web scraping framework designed for large-scale and complex website data extraction. It offers high-level abstractions, built-in data extraction tools using XPath and CSS selectors, asynchronous processing for parallel requests, and flexible pipelines for data storage, making it ideal for efficient and scalable web scraping projects.
This article walks through a real Python web‑scraping problem, explains why missing request headers cause failures, and provides complete, runnable code snippets—including header setup, cookie handling, and request loops—to retrieve stock data from Xueqiu, followed by a concise summary of the solution.
This guide demonstrates how to use PHP's cURL library to programmatically log into a website by configuring the login URL, form fields, request options, handling errors, and processing the response for tasks such as data scraping or automated testing.
This article walks through a Python solution for automatically solving web‑page captchas by capturing the image with Selenium, sending it to the Chaojiying service, and retrieving the decoded text, complete with sample code and execution results.
This article explains how to install php-webdriver via Composer, set up a Selenium WebDriver instance in PHP, and write a script that automates a Chrome browser to scrape search results from Baidu, demonstrating key WebDriver APIs for element interaction and data extraction.
This tutorial explains how to collect 2022 county‑level GDP and public budget data from the Chinese National Bureau of Statistics using Java's Jsoup library, transform the HTML tables into structured Excel files with EasyExcel, and provides complete source code and step‑by‑step analysis.
This article walks through a Python-based web scraper that extracts job listings from 51job.com, detailing required header and cookie settings, pagination logic, data parsing, and CSV output, and includes full code snippets and tips for handling site changes and further data analysis.
This article walks through a common Python web‑scraping issue where missing request parameters cause failures, demonstrates how to add the necessary headers—including User‑Agent and Origin—and provides complete, runnable code snippets that retrieve the expected data from the 10086 shop API.
This article demonstrates how to create a Python-based tool that crawls click‑event data from an e‑commerce site, processes and visualizes the information to verify event‑tracking code, covering preparation, data collection, analysis, and visualization steps with complete source code.
This tutorial demonstrates how to crawl the DouTuBa emoji website using Python, extract image URLs with regular expressions and BeautifulSoup, and download tens of thousands of images efficiently through a multithreaded downloader.
This article walks through a real‑world Python web‑scraping case, showing how to retrieve disease‑pest data from the Shandong government site using custom headers, cookies, and request parameters, and provides two complete code examples that successfully return the desired JSON results.
This article presents ten practical Python automation scripts—including HTML parsing, QR code scanning, screenshot capture, audiobook creation, PDF editing, StackOverflow querying, mobile device control, CPU/GPU temperature monitoring, Instagram uploading, and video watermarking—to help readers eliminate repetitive tasks and streamline their workflows.
This article walks through a Python web‑scraping solution for extracting and downloading audio files from a WeChat page, explains the problem, shows the implementation steps with screenshots, and provides the complete, ready‑to‑run code.
This guide shows how to collect nationwide weather data for the Qingming Festival via the free QWeather API, process it with Python to generate a CSV file, and visualize the rainfall distribution across China using a desktop data‑visualization tool.
This article introduces QueryList, a modern PHP content‑scraping tool that uses CSS selectors instead of regex, explains its two versions (V3 and V4), shows how to install it via Composer, demonstrates basic crawling code and various collection methods such as flatten, take, reverse, filter, map, and multi‑request concurrency.
This article walks through a common Python web‑scraping issue, explains why downloads may fail, and provides step‑by‑step solutions using the you‑get tool, including code snippets and troubleshooting tips to help beginners resolve network and firewall obstacles.
This article walks through building a Python web scraper that fetches stock trading data using the requests library and pandas, showing the complete code, how to set headers and cookies, and the resulting DataFrame, while highlighting the limits of relying solely on ChatGPT‑generated snippets.
This article walks through a real‑world Python Selenium web‑scraping issue, explains common driver configuration problems, and provides the exact code fix to help developers quickly get their web crawlers running smoothly.
This article walks through a Python web‑scraping challenge involving encrypted parameters, demonstrates a Selenium‑based solution with full code, shares community tips that fix the request payload, and shows how the issue was resolved step by step.
This tutorial demonstrates how to use Python's requests library to fetch video data from the YY Live API, parse the JSON response, extract video URLs, and programmatically download and save the videos, including code snippets for single and batch downloads.
This article walks through a Python web‑scraping solution that fetches GDP data from a website using the requests library, parses HTML with lxml, and demonstrates two approaches—manual XPath extraction and a streamlined pandas.read_html method—while providing complete code snippets and tips for handling pagination and data storage.
This article explains how a Python web‑scraping request failed due to extra nested parameters, shows the step‑by‑step debugging process, and demonstrates that deleting trivial 0/1 values can restore the request, offering a practical tip for similar crawling issues.
This article introduces the Scrapeasy Python library, showing how a single line of code can scrape websites, retrieve links, images, PDFs, and videos, and download them to local storage, making web data extraction fast and straightforward.
Discover ten practical Python automation scripts—from HTML parsing and QR code scanning to screenshot capture, PDF editing, StackOverflow querying, mobile device control, hardware monitoring, Instagram uploading, and video watermarking—that streamline repetitive tasks and boost productivity across diverse workflows.
An in‑depth guide shows how to decode URL‑encoded strings encountered during Python web scraping, explains the difference between two encoding formats, and provides ready‑to‑run urllib code that prints the original Chinese characters, helping developers troubleshoot similar crawling issues.
This article walks through a real Python Selenium web‑scraping issue, shows the error screenshots, explains why the script fails when the page isn’t fully rendered, and provides a reliable code solution along with practical tips for avoiding similar problems.
This article explains how to build a Python web crawler that fetches HTML pages with urllib, parses them using BeautifulSoup to extract image URLs, and downloads the images, covering all three stages with complete code examples and parser options.
This article explains how to use Python's requests and BeautifulSoup libraries to crawl Divnil.com, extract image detail page URLs, parse each page for high‑resolution image links, and download the images into a local folder with a complete, runnable script.
This tutorial walks through building a simple Python web crawler for the QDaily website, covering target analysis, environment setup, SQLite database creation, data extraction with requests and BeautifulSoup, storing articles and comments, performing basic analysis, and visualizing results with a word cloud.