Obscura is a Rust‑based headless browser that uses only 30 MB of memory and loads pages in 85 ms, offering built‑in anti‑detection, tracker blocking, zero external dependencies, and full Puppeteer/Playwright compatibility for AI automation and security‑research workloads.
The author refactors the z‑skills suite by separating video link discovery (z‑web‑pack) from actual downloading (z‑video‑downloader), explains why the split solves performance and reliability issues, details four concrete code and documentation changes, and shows a simple usage pattern for the updated skills.
Firecrawl is a 139K‑star open‑source web‑scraping API that handles dynamic JavaScript pages, full‑site crawling, search, and interactive browsing, offers built‑in proxy rotation and LLM‑ready Markdown/JSON output, and provides detailed code samples and deployment guides that outperform traditional tools like Scrapy and Selenium.
The article analyzes why acquiring public social‑media data is the biggest obstacle for AI‑driven content apps, compares DIY crawlers with commercial solutions, and demonstrates how RedFox’s unified API lets developers fetch multi‑platform data with a single key, minimal code, and lower maintenance costs.
This article explains how to construct a personal knowledge base using the author’s open‑source web‑pack Skill, which automates raw material collection, image localization, link expansion, and structured output, addressing the limitations of Obsidian’s Web Clipper and aligning with Karpathy’s LLM Wiki three‑layer architecture.
This article demonstrates how to use Python to crawl 7,275 Douban short reviews of the film “Letter to Grandma”, clean the data, generate a word‑cloud and frequency bar chart, and perform sentiment analysis that reveals over 91% of comments are positive.
Scrapling is an open‑source, adaptive web‑scraping framework that automatically tracks element changes, bypasses Cloudflare and other anti‑scraping defenses, offers multiple fetchers (including stealth mode), and delivers extraction speeds up to 784× faster than BeautifulSoup (BS4) while supporting concurrency, AI integration, and easy CLI usage.
Obscura, an open‑source Rust headless browser designed for AI agents and large‑scale crawling, cuts memory usage by 85%, reduces binary size, speeds page loads to 85 ms, starts instantly, includes strong anti‑detection features, and works with the Chrome DevTools Protocol as a drop‑in replacement for Headless Chrome.
This article explains how to turn simple API calls into robust, noise‑filtering tools—Search, Visit, Scholar, and Python—by adding domain blacklists, relevance scoring, query‑driven extraction, safety sandboxes, and a unified registry, ultimately boosting the success rate of LLM‑driven research agents.
Lightpanda, an open‑source headless browser written in Zig, claims up to eleven times faster execution and one‑ninth the memory of Chrome, offering a lightweight architecture for large‑scale web scraping, AI agent testing, and automation with detailed benchmarks, feature lists, usage commands, and known limitations.
This article provides a comprehensive guide to using the Go library chromedp for Chrome DevTools Protocol‑driven browser automation, covering its core architecture, practical techniques for navigation, network interception, anti‑bot evasion, performance tuning, and production‑grade recommendations with full code examples.
This guide explains how OpenClaw’s built‑in browser feature and the optional Browser Relay extension enable effortless data extraction from both public and login‑required websites, providing step‑by‑step installation commands, Chrome extension loading instructions, and a practical example of scraping top posts from a social platform.
This tutorial shows how to encode a JD.com search URL, fetch the page, and use XPath expressions in Python to reliably extract product names, links, images, and prices, offering a clearer alternative to regular‑expression or BeautifulSoup scraping.
This tutorial shows how to build a JD.com search URL, encode keywords, fetch the page with Python's urllib, and extract product details using regular expressions, providing code snippets, regex explanations, and sample output for beginners.
This article reviews three recently popular open‑source AI tools—Spider_XHS for Xiaohongshu data scraping, PPTAgent for one‑click presentation generation, and Code2Video for code‑driven video creation—detailing their key features, quick‑start instructions, and GitHub repositories, all free and ready to deploy.
This article walks through the complete design and implementation of a daily news summarizer, covering source selection, web‑scraping with BeautifulSoup, database schema with SQLModel, LLM‑based summarization, FastAPI endpoints, front‑end layout, category/date browsing, and a scheduled update loop.
The article examines the open‑source ai‑goofish‑monitor project, which combines Playwright‑driven browsing with large‑language‑model analysis to continuously scan Xianyu listings, filter out junk, and highlight high‑quality items, while also discussing its AI‑generated code, benefits, limitations, and security risks.
The "Academic Path" lecture series at Tsinghua University taught non‑CS students practical Python techniques—including Requests, AJAX, Selenium, PDF parsing, and multiprocessing—to extract, process, and analyze financial data, culminating in hands‑on case studies of ESG news, A‑share reports, and IFC databases.
ChromePHP is a PHP library that integrates the Chrome DevTools protocol, allowing developers to control Chrome or Chromium directly from PHP code for tasks such as navigation, JavaScript execution, screenshots, PDF generation, and input simulation, with simple Composer installation and flexible configuration.
Spider_XHS is an open‑source Xiaohongshu crawler that automates note, user, comment, and message extraction, offers watermark‑free media downloads, exports structured Excel/JSON data, integrates with the creator platform, and includes proxy and anti‑ban features, enabling marketers and researchers to cut weeks of manual work into hours.
This article walks through the end‑to‑end automation of a company’s daily dashboard report, covering reverse‑engineered login, API data extraction, image generation with canvas, OSS upload, and scheduled DingTalk webhook delivery, all implemented in Node.js.
This article introduces the Scrapeasy Python library, shows how to install it, initialize website and page objects, retrieve links, images, videos, and other file types, and download them with simple one‑line commands, providing a fast way to extract data from any site.
This guide shows how to use Python, Selenium, and requests to log into QQ Space, retrieve authentication cookies, compute the required g_tk token, fetch the full friend list via QQ's API, identify users who have blocked you, and optionally blacklist them automatically.
This tutorial demonstrates how to use Python and Selenium to collect thousands of hot comments from QQ Music, parse the data from dynamically loaded list items, store it in CSV files, and finally generate a word‑cloud visualization of the comment content.
This guide explains how to use industry sector classification to support value investing by scraping sector and stock data, parsing JSON responses, filtering by keywords, and exposing the results via FastAPI services, then displaying them in a Uniapp front‑end table for easy stock selection.
This article introduces DrissionPage, a Python library that merges Selenium and Requests, explains its three page objects, highlights seamless mode switching, built‑in utilities, API changes in version 4.0, and provides practical code examples for web automation, data crawling, and testing.
This article walks through a common Chinese character garbling problem in Python web crawlers, explains why automatic encoding detection can fail, and provides clear code examples—including manual GBK setting and re‑encoding tricks—to reliably extract readable text.
This article explores Browser-Use, an AI‑driven browser automation framework that combines large language models, visual perception, and DOM analysis to enable intelligent, multi‑step web tasks such as registration, price comparison, form filling, and monitoring, while detailing its architecture, historical context, core modules, and future challenges.
Discover how AI, especially large language models, transforms traditional web scraping by introducing semantic understanding, dynamic adaptability, and automated extraction, with in‑depth reviews of emerging tools like Crawl4AI and Browser‑use, practical code examples, best‑practice guidelines, and deployment tips for modern data collection.
This article presents ten practical Python automation scripts—covering HTML parsing, QR code scanning, screenshots, audiobooks, PDF editing, StackOverflow queries, mobile control, temperature monitoring, Instagram uploads, and video watermarking—to help you streamline daily repetitive tasks efficiently.
This article walks through a real‑world Python web‑scraping problem, shows why the original XPath selector fails, provides corrected code with a working XPath expression, and highlights best practices such as adding request headers for reliable crawling.
This article walks through a Python web‑scraping issue, shows how to clean scraped data using regular expressions, provides sample code for extracting both floating‑point and integer numbers from mixed strings, and offers a concise solution that readers can apply to their own projects.
Crawl4AI is an open‑source AI agent that automates web crawling and data extraction, offering free usage, intelligent parsing, structured JSON/Markdown output, and versatile features like scrolling, multi‑URL scraping, media and metadata extraction, all demonstrated through step‑by‑step Python examples and integration with AI agents.
This article walks through a Python web‑scraping problem, explains why the original script produced incorrect results, presents a corrected implementation with full code examples, and clarifies the underlying bug caused by mutable data reuse.
Discover eight essential Python libraries—Rich, Typer, Pendulum, Pydantic, Faker, tqdm, Requests‑HTML, and Loguru—that transform logging, CLI creation, date handling, data validation, mock data generation, progress tracking, web scraping, and logging, showing how leveraging existing tools can dramatically increase development efficiency.
This article walks through a Python Selenium web‑scraping issue raised in a community chat, shows the error screenshots, explains two resolution methods—including version rollback and driver compatibility—and provides a ready‑to‑run code snippet to help readers overcome the problem.
Discover how using powerful Python libraries like Rich, Typer, Pendulum, Pydantic, Faker, Tqdm, Requests‑HTML, and Loguru can replace repetitive code, streamline debugging, data handling, CLI creation, and logging, letting you focus on real value creation.
This article walks you through using Playwright in Python to scrape the Xinfadi market website, showing how to capture URLs, handle request payloads, extract JSON data, and process results with a complete, runnable code example.
This tutorial walks through using Python to crawl Weibo, covering environment setup, three login methods, data extraction functions for user info, posts and comments, anti‑crawling strategies, storage to CSV or MySQL, a full example script, and legal considerations.
This article reviews six open‑source MCP servers—Bright Data, Graphiti, GitIngest, Terminal, Code Executor, and MindsDB—showing how each extends large language models with web scraping, long‑term memory, code navigation, command‑line control, sandboxed Python execution, and multi‑source data integration, complete with practical code examples.
This article provides a comprehensive overview of web‑scraping technologies, compares popular tools such as requests, BeautifulSoup and Selenium, introduces AI‑assisted crawling with DeepSeek, and walks through practical steps for using BrightData’s platform to collect industry data, complete with ready‑to‑run Python code.
This tutorial walks you through building a Python web‑scraper for Beike (Lianjia) second‑hand property listings, covering session spoofing, dynamic parameters, pagination, multithreaded detail fetching, data cleaning, and exporting results to Excel in a step‑by‑step manner.
Learn how to efficiently scrape web pages using Python by exploring multiple approaches—including simple requests with BeautifulSoup, fast parsing with lxml, dynamic content extraction with Selenium, and large‑scale crawling with Scrapy—complete with installation steps, code snippets, and detailed explanations.
This article walks through using Pandas to directly read HTML pages, extract table data, handle AJAX‑loaded JSON and CSV formats, and save results, providing concise code examples and visual steps for effective Python web scraping and data mining.
This article introduces ten practical Python-based web scraping methods—including requests, Selenium, Scrapy, Crawley, PySpider, aiohttp, asks, vibora, Pyppeteer, and Fiddler‑plus‑Node reverse engineering—explaining their use cases, advantages, and code examples for efficiently gathering e‑commerce and app data.
This article outlines Python's advantages for web crawling, introduces core libraries such as Requests, BeautifulSoup, and Scrapy, details a step-by-step development workflow, provides practical code examples for extracting news titles, and highlights important considerations and advanced techniques for robust scraper implementation.
This article demonstrates how to use Python's requests and BeautifulSoup libraries to scrape current and 14‑day weather data from China Weather, store the results in CSV files, and perform comprehensive visual analysis—including temperature, humidity, AQI, wind direction, and forecast charts—using pandas, numpy, and matplotlib.
This tutorial walks through practical jsoup usage within Spring Boot 3, covering dependency setup, parsing HTML from strings, fragments, URLs or files, extracting titles, links, images, applying CSS selectors, modifying elements, and sanitizing content to prevent XSS attacks.
This article explains how to use XPath in Python to skip the first table header row during web scraping, provides a concise code example, and discusses alternative approaches, helping readers efficiently extract desired list items from HTML structures.
This article walks through using Python's pyquery library to fetch the names and URLs of popular songs from NetEase Cloud Music, compares it with previous xpath and BeautifulSoup approaches, provides complete runnable code, and highlights key challenges in selector usage.
This article explains why XPath fails on NetEase Cloud Music pages, demonstrates a regex‑based Python solution with full source code, shows sample output, and outlines future tutorials using XPath, BeautifulSoup, and PyQuery to strengthen Python selector skills.
Learn how to extract song names and URLs from NetEase Cloud Music's hot tracks page using Python's requests library and XPath selectors, including handling malformed HTML, code examples, and tips for replacing interfering tags to ensure reliable scraping.
This tutorial walks through building a Python script that fetches a QQ Music QR code, displays it locally, monitors its status, extracts required encrypted parameters from cookies, and completes the login process by repeatedly polling the authentication endpoint to obtain a session.
This tutorial walks you through creating a Python script that crawls an online dictionary to fetch word‑meaning pairs, formats them, and automatically emails the list each day, using libraries like lxml, requests, BeautifulSoup, and smtplib for scheduling and delivery.
This article demonstrates how to reverse‑engineer a JavaScript encryption function used to hide video URLs, translating the logic into Python code step by step, and shows how to obtain the final downloadable link via web scraping, complete with code snippets and example outputs.
Learn how to use Python to automate file management, web scraping, email sending, spreadsheet processing, and scheduled tasks, saving time and reducing repetitive work with concise code examples.
This article demonstrates how to combine Playwright web‑scraping of multiple Chinese movie platforms with the DeepSeek AI model to automatically collect data and generate a scientific prediction of the box‑office revenue for the film "Ne Zha 2".
This article walks through a real‑world case of reverse‑engineering a JavaScript video‑encryption routine, translating its core functions into Python, and demonstrates how to reconstruct the encrypted URL to download the video using a web‑scraping approach.
This article explains what constitutes high‑quality LLM training data, why large datasets are essential, outlines the step‑by‑step process for collecting, preprocessing, and fine‑tuning models, and highlights the best data sources—including web content, books, code repositories, and news—while noting available free datasets.
This article demonstrates how to use Python's requests library and regular expressions to scrape Baidu search result titles and URLs, save them to a CSV file, and suggests alternative parsers like XPath or BeautifulSoup for further extraction.
Learn how to scrape Baidu search result titles and URLs using Python's requests library and lxml's xpath parsing, complete with a ready-to-run script, CSV output, and step-by-step explanation that builds on earlier regex and BeautifulSoup methods.
This tutorial explains e‑commerce web scraping fundamentals, covering definitions, tool types, data categories, step‑by‑step Python script creation with Requests, BeautifulSoup, and Selenium, provides sample code for Amazon, Walmart, and eBay, discusses challenges like dynamic pages and anti‑scraping measures, and recommends using specialized scraping APIs.
This article introduces web crawling fundamentals, demonstrates how to inspect HTML elements, walks through simple examples using urllib, requests, and BeautifulSoup, and culminates in a complete Python script that extracts chapter links and contents from a novel website, saving them to a text file.
This article compares Python and PHP for web scraping, outlining each language's strengths, ecosystem, performance, learning curve, and community support to help readers decide which tool best fits their project requirements and experience level.
This guide explains the essential proxy concepts for Python web scraping, covering what proxies are, common types, how they protect crawlers, practical usage with the requests library, and the role of proxy pools in improving scraping efficiency.
This tutorial demonstrates how to analyze a fund ranking website, construct dynamic URLs, extract six‑digit fund codes, crawl fund holding pages with requests and Selenium, store the results in MongoDB, and finally process the data to identify the most frequently held stocks across thousands of funds.
This article presents a collection of practical Python scripts covering file management, web scraping, email sending, Excel handling, data cleaning, image processing, system monitoring, PDF manipulation, OCR, database interaction, social media posting, testing, and cloud storage, each with clear descriptions and ready‑to‑run code examples.
This article walks through a practical Python solution for parsing a list of courier records, extracting province information, grouping entries by location using dictionaries, and optionally applying regular expressions and pandas for further analysis, all illustrated with clear code examples and output screenshots.
This article introduces eleven powerful Python libraries for web scraping—including Requests, BeautifulSoup, Scrapy, Selenium, PyQuery, Lxml, Pandas, Pyppeteer, aiohttp, Faker, and ProxyPool—explains their key features, provides ready‑to‑run code snippets, and demonstrates a real‑world news‑site crawling case study.
This article demonstrates how to use Python's requests and BeautifulSoup libraries to scrape current and 14‑day weather data from China Weather Net, store the results in CSV files, and visualize temperature, humidity, air quality, and wind patterns using Matplotlib, NumPy, and Pandas.
This article walks through a real‑world Python web‑scraping issue where Chinese characters appear as garbled text, explains why the default encoding fails, and shows how setting the response’s apparent encoding resolves the problem, complete with sample code and practical tips for posting questions.
This article explains the PHP get_headers() function, its syntax, parameters, and usage examples, demonstrates how to retrieve and display HTTP response headers from a URL, and discusses common application scenarios such as checking remote file existence, obtaining file metadata, and supporting web crawling or monitoring.
This article presents a Python‑based project that continuously crawls Weibo hot‑search data, stores it with timestamps, and visualizes trends through dynamic bar, line, and word‑cloud charts using libraries such as BeautifulSoup, pandas, schedule, pyecharts, and jieba.
This article provides a concise Python cheat sheet covering common file and directory operations, CSV/JSON data handling, web request and crawling techniques, automation tasks such as scheduling and email sending, and various string manipulation utilities, each illustrated with ready‑to‑run code snippets.
Learn step‑by‑step how to extract text from a CSV generated by a Python web crawler, clean it with stop‑words, create a word‑cloud visualization, compute word frequencies, and perform sentiment analysis using jieba and SnowNLP, with all code snippets provided.
This article walks through a Python web‑scraping project that fetches skin images, names, and prices from Honor of Kings, covering project background, required tools, site analysis, code implementation, data processing, and legal considerations.
This article explains how to build a Python script that searches a comic website, extracts chapter links and image URLs using requests and BeautifulSoup, and downloads the images into organized folders with multithreaded support, while outlining required modules and potential improvements.
This guide walks you through extracting novel chapters from the Biquw.com website with Python, explaining how to obtain the book ID, retrieve the chapter list, download each chapter using requests and BeautifulSoup, handle common anti‑scraping measures, and organize the downloaded files.
This tutorial walks through creating a Coze bot that fetches a Juejin author's information and articles via web scraping, processes the data to generate a concise author profile, identifies expertise domains, ranks hot articles, and finally publishes the bot for interactive use.
This article introduces seven powerful Python libraries—Pandas, Selenium, Flask, Scrapy, Requests, Faker, and Pillow—explaining their core features, typical use cases, and providing ready‑to‑run code snippets to help developers boost productivity and automate routine tasks.
This article presents 17 versatile Python scripts that automate common tasks such as file management, web scraping, email handling, Excel processing, database interaction, system administration, and more, offering code examples and explanations to help developers and analysts improve productivity.
This article explains a Python web‑scraping issue where Selenium sometimes returns only previously scraped rows after page navigation, and provides a step‑by‑step solution using requests, pagination parameters, regex extraction of IDs, and PDF download handling to reliably collect data.
This article walks through a Python community member's problem of turning raw HTTP header strings and cookies into clean dictionary objects, explains why newline characters cause issues, and provides step‑by‑step code solutions using string replacement, JSON conversion, and built‑in libraries.
This article demonstrates how to use Python to scrape Weibo comments about the TV series 'The Legend of the Year', generate word clouds, compute character frequencies and co‑occurrences, and visualize the resulting character relationship network with Gephi.
This tutorial demonstrates how to install Selenium, log into QQ Space, and programmatically retrieve both status updates and album photos using Python's Selenium and BeautifulSoup libraries, including detailed code examples for login, scrolling, and image downloading.
This article presents six practical Python techniques—including exception handling, list comprehensions, regular expressions, web scraping with BeautifulSoup, lambda functions, and simple data visualizations using matplotlib—to help developers write cleaner, more efficient code and enhance their programming workflow.
This article explains how the PHP cURL library, especially the curl_exec() function, can be used to initialize a session, set options, execute HTTP requests, handle errors, and retrieve remote page content efficiently.
This article walks through a Python web‑scraping solution for downloading individual files from a site, presenting the problem, multiple approaches, and a complete, ready‑to‑run script, while also offering tips for handling large data and avoiding IP blocks.
The article showcases a variety of Python scripts that automate tasks such as subtitle and video downloading, IMDb data extraction, email notifications, PNR status checks, WhatsApp avatar monitoring, face recognition for photo sorting, and a custom DC++ sharing tool, providing code examples and usage details for each utility.
This article reviews the ten most essential Python libraries for data science, covering data acquisition, analysis, machine learning, and visualization, and provides concise code examples to help beginners quickly start using tools like Beautiful Soup, NumPy, pandas, scikit‑learn, TensorFlow, Keras, Matplotlib, and seaborn.
This article explains how to obtain cookies from an initial HTTP response using Python's requests library, reuse them in subsequent requests, and handle related security considerations such as CSRF tokens and robots.txt compliance, providing a complete code example and step‑by‑step guidance.
This article explains how combining AI with Node.js crawling through the x‑crawl library makes web data collection more intelligent, flexible, and resilient to website changes, and provides detailed feature descriptions, code examples, and practical usage scenarios.
This article walks through solving a Pandas data‑analysis problem raised in a Python community, showcasing multiple Python techniques—including string manipulation, zero‑width spaces, and slicing—to clean and transform data, complete with code snippets and result screenshots.
This article demonstrates how to use Python's requests, lxml, and pandas libraries to crawl GDP data from a website, parse the HTML tables, and efficiently write the extracted rankings, regions, GDP values, and years into a CSV file, providing a complete, runnable example for web scraping beginners.
This article presents a collection of useful Python scripts—including file sorting, empty‑folder removal, batch renaming, web‑scraping, bulk image downloading, form automation, and various text‑processing utilities—each accompanied by clear explanations and ready‑to‑run code examples.
This article explains how to overcome JavaScript-rendered web pages when scraping with Python, comparing tools like Selenium, Playwright, and the DrissionPage library, and provides a concrete code example that fetches and prints page HTML, along with practical tips for handling JSON data sources.
This article explains why Chinese characters often appear garbled when using Python web crawlers and presents three practical solutions—switching to response.content, manually setting the response encoding, and applying a universal encode‑decode trick—to reliably decode GBK‑encoded text.
This article demonstrates how to use Jsoup to fetch and parse HTML content from a web page, extract specific table data via CSS selectors, map the data to Java objects, and efficiently write the results to an Excel file with EasyExcel, highlighting memory advantages over POI.
This article explains why a Python web crawler often fails to retrieve JavaScript‑rendered page content, outlines common causes such as dynamic loading, anti‑scraping measures, and server‑side rendering, and offers practical techniques to capture the fully rendered HTML.
This article presents nine practical Python automation scenarios—including file management, web scraping, text processing, email handling, Excel manipulation, database interaction, social media posting, system task scheduling, and image editing—each accompanied by clear script examples and explanations to help readers streamline repetitive tasks efficiently.
