Tagged articles
anti-scraping
25 articles · Page 1 of 1
Scrapling is an open‑source, adaptive web‑scraping framework that automatically tracks element changes, bypasses Cloudflare and other anti‑scraping defenses, offers multiple fetchers (including stealth mode), and delivers extraction speeds up to 784× faster than BeautifulSoup (BS4) while supporting concurrency, AI integration, and easy CLI usage.
PythonScraplingadaptive crawler
0 likes · 16 min read

Learn how to overcome anti‑scraping defenses by using Python's requests_html library to fetch and parse dynamic web pages, with a complete code example that extracts company names from a target site, plus tips on handling cookies, headers, and Unicode decoding.
anti-scrapingdata-miningrequests-html
0 likes · 6 min read

This article traces the development of web crawlers from their 1990s origins to modern implementations, examines their multifaceted value in search, data analysis, and archiving, outlines technical, ethical, and legal challenges for both crawler creators and target sites, and presents practical strategies to mitigate malicious crawling.
SecurityWeb Crawlinganti-scraping
0 likes · 24 min read

This guide explains how unemployed developers can use Python, especially web‑scraping techniques, to secure freelance gigs by leveraging various platforms, community groups, and effective order‑taking strategies while warning about common pitfalls and anti‑scraping challenges.
Freelanceanti-scrapingdata processing
0 likes · 7 min read

The article explains several typical situations—such as not configuring HTTPS proxies, using server IPs, non‑anonymous proxies, polluted IP pools, and lack of HTTP/2 support—that cause websites to easily detect that a request is made through a proxy, even for beginner Python scrapers.
HTTPProxyPython
0 likes · 7 min read

This article walks through a real‑world Python web‑scraping case, explaining why standard requests failed, how the requests_html library can overcome anti‑scraping defenses, and provides a complete, runnable code example with analysis and results.
anti-scrapingrequests_html
0 likes · 6 min read

This article walks through decoding custom font anti‑scraping on a Chinese group‑buying site using Python, covering page fetching, CSS font URL extraction, .woff download, glyph mapping with fontTools, OCR via ddddocr, DOM manipulation with BeautifulSoup, and exporting the cleaned data to Excel.
BeautifulSoupPythonanti-scraping
0 likes · 28 min read

This guide explains common anti‑scraping defenses—identity verification via request headers and IP rate limiting—and shows how to bypass them in Python using custom user‑agents, request throttling, and BeautifulSoup to successfully scrape Douban’s Top 250 movies.
anti-scraping
0 likes · 6 min read

This article explains how to overcome common anti‑scraping defenses such as identity verification and IP rate limiting by spoofing the User‑Agent header and adding request delays, providing complete Python code examples using requests and BeautifulSoup to scrape Douban's Top 250 movies.
BeautifulSoupIP throttlingUser-Agent
0 likes · 6 min read

This tutorial explains how to analyze the Mayi short‑rent website, overcome its anti‑scraping defenses by setting appropriate Cookie and User‑Agent headers, and use Python's urllib2 and BeautifulSoup to extract rental details, store them in CSV, and optionally employ Selenium.
BeautifulSoupCookiePython
0 likes · 8 min read

This article explains how to disguise a web crawler's identity by customizing request headers, managing request frequency with sleep and proxy settings, and tackling common anti‑scraping mechanisms such as captchas, dynamic loading, and encrypted content using tools like Selenium.
anti-scrapingproxiesrequest headers
0 likes · 6 min read

This tutorial walks you through creating a Python web scraper that extracts images from a target site, handles anti‑scraping measures, saves them into categorized folders, and logs the download results, while explaining the required libraries, code structure, and best practices.
Image DownloadPythonXPath
0 likes · 6 min read

This tutorial explains how to crawl 58.com job listings with Python, extract location, company, and salary information, handle anti‑scraping defenses using realistic headers and random User‑Agents, and save the results into a text file.
Pythonanti-scrapingdata mining
0 likes · 7 min read

This article explains how many Chinese web sites use hidden CSS ::before content to hide characters, shows how to locate the relevant network request, decode the span class mappings from obfuscated JavaScript, and restore the original text for successful web scraping.
CSSJavaScriptWeb Crawling
0 likes · 10 min read

This article explains the concept and benefits of CSS sprites, analyzes their drawbacks for web performance and security, and provides a step‑by‑step Python‑based method—including code snippets—to extract and sum numbers hidden behind sprite images used as an anti‑scraping measure.
CSSFront-endSecurity
0 likes · 9 min read

This tutorial explains how to overcome common anti‑scraping defenses of a proxy‑listing website by capturing legitimate HTTP headers with Fiddler, configuring the Python requests library, and building a dynamic proxy pool to keep your crawler running smoothly.
FiddlerProxyPython
0 likes · 5 min read

This article explains how to hide real intentions behind visible actions by using text obfuscation and custom Base64‑like encoding to defeat standard web scrapers, detailing the underlying principles, decoding challenges, and Python implementations of a flexible Custom64 encoder.
Base64Pythonanti-scraping
0 likes · 10 min read

This guide explains practical techniques to avoid IP bans and 403 errors when web‑scraping, covering explicit and implicit waiting, User‑Agent spoofing, proxy usage, IP pools, and distributed crawling architectures.
ProxyPythonanti-scraping
0 likes · 8 min read

This guide explains practical techniques such as faking User‑Agent headers, rotating cookies, adding random delays, and using proxy pools to prevent IP bans while crawling large amounts of data from websites with anti‑scraping defenses.
User-Agentanti-scrapingcookies
0 likes · 4 min read


21CTOSep 28, 2019 · Backend Development
This article walks through the challenges of scraping Dazhong Dianping, explains how the site hides numeric data with custom CSS fonts, and provides a complete Python workflow—including HTTP requests, font extraction, glyph rendering, and OCR—to decode and retrieve the protected information.
CSS encryptionOCRPython
0 likes · 13 min read

Learn how to start web scraping with Python by mastering the three core steps—fetching, analyzing, and storing data—using urllib and requests, handling login, evading anti‑scraping measures like user‑agents and IP proxies, and saving results to JSON, CSV, or MongoDB.
Data StoragePythonanti-scraping
0 likes · 9 min read


JD TechSep 7, 2018 · Information Security
The ISC 2018 conference highlighted the growing importance of big data and artificial intelligence security, presenting JD's research on anti‑scraping techniques, AI‑driven defenses against black‑market attacks, and a service‑oriented approach to protecting user data across enterprises.
AI securityanti-scrapingbig data
0 likes · 5 min read


ITPUBMay 2, 2017 · Backend Development
This guide explains why websites employ anti‑scraping defenses, outlines the most common header checks such as User‑Agent, Referer, and Cookies, and provides practical Scrapy code snippets for rotating user agents, managing proxies, handling X‑Forwarded‑For, limiting request rates, and dealing with dynamic AJAX content using Selenium or PhantomJS.
HeadersProxyanti-scraping
0 likes · 7 min read


ITPUBMar 21, 2016 · Backend Development
This guide outlines the main anti‑scraping techniques used by websites—including header validation, user‑behavior monitoring, and dynamic content loading—and provides practical methods such as header spoofing, IP proxy rotation, request throttling, and Selenium/PhantomJS automation to overcome them.
HeadersPhantomJSanti-scraping
0 likes · 6 min read


21CTOJan 26, 2016 · Backend Development
This article summarizes common anti‑scraping techniques—including header checks, user‑behavior detection, and dynamic page defenses—and provides practical ways to circumvent them using custom headers, IP proxies, request timing, and tools like Selenium with PhantomJS to simulate real browsers.
HeadersProxyanti-scraping
0 likes · 6 min read
