This tutorial walks through using Python to crawl Weibo, covering environment setup, three login methods, data extraction functions for user info, posts and comments, anti‑crawling strategies, storage to CSV or MySQL, a full example script, and legal considerations.
The article details Bilibili's comprehensive anti‑crawling solution, covering the problem background, a two‑layer detection framework integrated with APIGW and GAIA, risk perception, strategy iteration, verification mechanisms, quantitative results, and future improvement directions, all illustrated with concrete examples and performance numbers.
This article details Bilibili's comprehensive anti‑crawling system, covering the background of API abuse, the data‑flow framework, risk perception, strategy iteration, verification mechanisms, gateway signing design, and the measurable impact on normal and special‑case interfaces.
Bilibili combats API abuse by deploying a two‑layer anti‑crawling system—gateway‑side signature verification and a GAIA risk‑control engine integrated into APIGW—that unifies device data, applies flexible rule packages, triggers diverse human challenges, and has already blocked billions of malicious requests with over 85% recall while preventing service outages.
This article explains the technical principles and implementation steps of web crawlers, introduces common crawling frameworks, provides a Python example for extracting app store rankings, and then details various anti‑crawling methods such as CSS offset, image camouflage, custom fonts, dynamic rendering, captchas, request signing, and honeypots, followed by counter‑strategies for each.
The article explains web‑crawling basics, Python and Scrapy examples, then surveys common anti‑crawling defenses such as CSS offsets, image camouflage, custom fonts, dynamic rendering, captchas, request signatures and honeypots, and finally presents anti‑anti‑crawling countermeasures—including CSS‑offset reversal, font decoding, headless‑browser rendering and YOLOv5‑based captcha cracking, while stressing legal compliance.
The article outlines several anti‑crawling techniques—including IP restrictions, User‑Agent validation, CAPTCHAs, AJAX loading, noscript tags, and cookie checks—while also offering practical advice for writing ethical, efficient, and robust web crawlers.
The article examines why web content needs protection, explains common server‑side header checks, describes client‑side JavaScript fingerprinting and headless‑browser detection methods, and outlines practical anti‑crawling measures such as CAPTCHAs and robots.txt, highlighting the ongoing cat‑and‑mouse game between crawlers and defenders.
This article explores the cat‑and‑mouse battle between crawlers and API endpoints, detailing how dynamic signatures, token‑based authentication, time‑bound hashes, rate‑limiting, and code obfuscation can be used to defend against scraping while also showing how attackers can reverse‑engineer and bypass these defenses.
The article outlines typical anti‑crawling measures such as browser detection, captchas, login requirements, JavaScript obfuscation, and behavior‑based blocks, and provides practical counter‑strategies including header spoofing, captcha solving, session/token handling, JS emulation, and human‑like request pacing.
This article introduces the fundamentals of web crawlers, typical crawling methods, and a comprehensive set of anti‑crawling strategies—including IP control, browser and device simulation, CAPTCHA cracking, and traffic analysis—while detailing the architecture and capabilities of the 58 anti‑crawling platform.
This article explains the evolution and classification of web crawlers, describes the design and components of the Mole distributed crawler—including scheduler, fetcher, processor, rate‑limiting, URL deduplication, and Elasticsearch storage optimization—and outlines common anti‑anti‑crawling strategies.
The article describes the goals, architecture, core functions, and key characteristics of a comprehensive anti‑crawling platform that systematizes strategy management, data cleaning, monitoring, and rapid response to protect APIs and improve data reliability for large‑scale online services.
The article explains anti‑crawling concepts, current challenges, classification of techniques (client‑side, middle‑layer, server‑side, real‑time vs. non‑real‑time), and argues for a systematic, platform‑driven approach to continuously adapt strategies against evolving web scrapers.
This article introduces web crawlers, classifies them by technology and intent, presents statistics on crawler traffic across industries and regions, and analyzes the various harms they cause, laying the groundwork for future discussions on anti‑crawling strategies.
This article explores why web content needs protection, explains common server‑side and client‑side anti‑crawling methods—including User‑Agent checks, token cookies, headless‑browser detection, fingerprinting, captchas, and robots.txt—and offers practical guidance for raising the cost of unauthorized scraping.
This article explores a variety of front‑end anti‑crawling techniques—from font‑face obfuscation and background‑image sprites to pseudo‑elements and iframe loading—illustrating how developers can make data extraction harder for bots while acknowledging that no method is foolproof.
The article explores the hidden, high‑traffic world of web crawlers and anti‑crawling measures, revealing why most online requests are bots, how companies decide to crawl or block, the technical and organizational challenges involved, and what the future may hold for this perpetual cat‑and‑mouse game.
This article explores the often hidden and contentious world of web crawling and anti‑crawling, detailing industry motivations, the massive proportion of bot traffic, the technical arms race between scrapers and defenders, and the broader impact on developers, companies, and security practices.
This article explores the hidden, often unglamorous world of web crawling and anti‑crawling, detailing why companies need these technologies, the massive traffic they generate, the technical arms race between crawlers and defenders, and the evolving strategies and challenges that shape the industry today.
This article examines the hidden, often unglamorous world of web crawling and anti‑crawling, revealing why companies deploy aggressive scraping and defensive measures, the technical arms race between crawlers and defenders, the impact on engineers' careers, and future trends in this contested space.
This article shares practical anti‑crawling concepts, classifications of crawlers, design principles, traditional and JavaScript‑based countermeasures, and operational trade‑offs, illustrating how Ctrip's hotel R&D team balances commercial protection with technical feasibility.
Web crawlers, especially AI‑driven ones, threaten site performance and data ownership, so this article reviews common anti‑scraping methods—from IP and header analysis to behavior detection—and reveals two unconventional defenses: data poisoning and a deposit‑based access model that penalize malicious bots.
