Scrapling: Adaptive Web Scraping That Survives Site Redesigns

Core Problem: Scrapers Break When Sites Redesign

The article opens with the central pain point: you write a scraper using a CSS selector like .title, then two months later the site renames it to .product-title and your script returns empty. At scale — hundreds of scrapers across dozens of machines — you don't know which broke, when, or whether data is missing or misaligned. The author compares this to putting sticky notes on 100 security cameras labeling "package locker here"; when the building renovates and moves lockers, every note is invalid.

A second hidden trap: anti-bot measures. Sites rate-limit IPs, serve CAPTCHAs, or return interstitial pages. You must masquerade as a real browser, and that disguise must stay in sync with actual browser versions — if your fingerprint says Chrome 148 while clients run 151, you're detected.

Scrapling's Solution: Three-in-One Library

Scrapling bundles fetching, parsing, and anti-bot evasion into one library, aiming to minimize the cost of "site redesigned" events. As of October 2024 v0.5, it has 95k GitHub stars and 910k monthly PyPI downloads.

Five Fetcher Tiers (Choose by Need, Not Version)

The library provides two main classes with five capability tiers, ordered by complexity:

Fetcher — plain HTTP requests (enhanced requests): fast, low memory, no browser needed. Cannot execute JavaScript.

Fetcher + fingerprint spoofing — same class, add parameter to mimic Chrome's TLS fingerprint and headers. Sufficient for most sites.

StealthyFetcher — launches headless Chromium via Playwright with heavy anti-detection: isolated JS environment, stripped Playwright fingerprints, canvas noise, WebRTC IP leak blocking, timezone spoofing. (v0.5 merged the old DynamicFetcher / DynamicSession into this.)

Cloudflare solving — StealthyFetcher with solve_cloudflare=True handles JS challenges, checkbox CAPTCHAs, and invisible challenges.

AI Agent / MCP Server — exposes 16 tools for Claude Code, Cursor, etc.

Documentation compares tiers: speed (tier 1: 5★, tier 5: 1★), stealth (tier 1: 2★, tiers 2-5: 5★), memory (tier 1: 1★, browser tiers: 3★). The rule: start with tier 1; only escalate if data isn't retrievable . 90% of cases need only tier 1. Common mistake: pip install scrapling[all] pulls Chromium for a static blog list. Minimal path: pip install scrapling (parser only), then pip install scrapling[fetchers] + scrapling install for browser bits.

Adaptive Element Matching: The Killer Feature

When a selector fails after a redesign, Scrapling's adaptive mode re-finds the element by stored "appearance" — not ID or class (which change first), but six stable traits:

Element's own tag, text, attributes (names & values)

Sibling tag names

Ancestor path (tags only)

Parent tag, attributes, text

It bets on "content unchanged, only shell swapped." Storage uses SQLite, isolated per domain ( a.com separate from b.com). Usage:

# First run: normal select, auto-save appearance
element = page.css('#p1', auto_save=True)

# After redesign: original selector fails
element = page.css('#p1', adaptive=True)  # finds it again

Validation example: 2010 StackOverflow button selector

#hmenus > div:nth-child(1) > ul > li:nth-child(1) > a

— 16 years later on modern StackOverflow, standard lookup fails; adaptive returns the same button.

Caveats documented: only the first matched element is saved; if a selector matches multiple positions, adaptive returns only the first. Two failure modes admitted: no match (check storage), or wrong match (write more specific selector or manual save with more context). If the site rewrites content entirely, adaptive fails — by design.

Parser: Familiar Syntax, Extreme Speed

Selector syntax mirrors Scrapy/Parsel: page.css('.quote'), page.xpath('//div[@class=quote]'), ::text, ::attr(href). Extras: find_by_text('Albert Einstein'), find_by_regex(r'\d+'), find_similar() — given one product card, grabs all similar cards in the same table by matching DOM depth, tag/parent/grandparent tags, then fuzzy attribute comparison (excludes href / src by default; threshold 0.2). Four-layer filter yields ~99% confidence.

Adaptive relocation threshold: 40% (v0.5+). Below that, returns empty with warning rather than a garbage match.

Elements behave as objects: .parent, .next, .below_elements, .attrib['href'], chaining like page.css('.quote')[0].css('.text::text').get().

Parser works offline: Selector(html_string) and Response (adds status/headers/cookies). Scrapy integration via @scrapling_response decorator — drop-in replacement for response without rewriting spider logic.

Benchmarks: Parsel-Level Speed, Orders Faster Than BS4

Official benchmark: 5,000 nested elements, text extraction. Results (100-run average on author's script):

Scrapling: 1.99 ms (1x baseline)

Parsel / Scrapy: 2.06 ms (~1.03x)

PyQuery: 23.98 ms (~12x)

Selectolax: 197.02 ms (~99x)

BeautifulSoup + lxml: 1562.1 ms (~785x)

BeautifulSoup + html5lib: 3412.73 ms (~1715x)

Scrapling ≈ Scrapy because it uses Parsel's translator. The gap vs BS4 isn't "better code" — it's lxml vs pure-Python traversal. Element similarity test: Scrapling 2.3 ms vs AutoScraper 12.58 ms (5.47x). JSON serialization uses orjson, claimed 10x stdlib.

Anti-Bot Layer: Deep Implementation Details

Beyond header spoofing, StealthyFetcher implements:

Canvas noise — randomizes pixel output per request to defeat canvas fingerprinting.

CDP & WebRTC blocking — prevents Runtime.enable automation detection and real IP leaks even with proxies.

Fingerprint version auto-sync — reads browsers.json from Playwright's patchright to align spoofed fingerprint with actual driven Chromium version. Code enforces MINIMUM_VERSION = 149 (Chrome), 147 (Firefox), 145 (Edge). Solves hardcoded-version rot.

TLS fingerprint (JA3) — via curl_cffi with impersonate='chrome', mimicking full handshake: cipher suites, extensions, curve order. Nine browser profiles: edge, chrome, chrome_android, safari, safari_beta, safari_ios, safari_ios_beta, firefox, tor. Versions update with curl_cffi.

Note: Scrapling doesn't implement JA3 calculation itself; it leans on curl_cffi 's built-in browser fingerprint library. No public JA3 pass-rate data exists in docs/issues.

Ad/tracker blocking — ~3,500 domains auto-blocked during browser fetch; shrinks HTML for downstream LLM token savings.

DNS leak protection — dns_over_https=True routes DNS via Cloudflare DoH.

Resource disabling — disable_resources=True drops fonts, images, media, stylesheets, websockets; ~25% faster, saves proxy bandwidth.

Crawler: Scrapy-Like with Practical Additions

Architecture: Spider → Scheduler (priority queue) → Crawler Engine (concurrency, delay, robots.txt, retry) → SessionManager (routes by sid). Defaults: concurrent_requests=4, concurrent_requests_per_domain=0 (unlimited), download_delay=0, robots_txt_obey=False (author's rationale: avoid accidental behavior). Conservative defaults favor polite scraping.

Added features:

AutoThrottle — per-domain adaptive delay: start 5s ( autothrottle_start_delay), max 60s ( autothrottle_max_delay). Non-2xx or is_blocked() doubles delay (0.5→1→2→4→8…). 429/503 respects Retry-After (seconds or HTTP-date), capped at 60s. Recovers speed when unblocked. Two documented gotchas: learned delays not checkpointed (restart resets to 5s); concurrent_requests_per_domain doubles as throttle target (unset → target 1), and throttled domains still occupy global concurrency slots.

robots.txt — optional ( robots_txt_obey=True), uses Protego, respects Disallow, Crawl-delay, Request-rate; takes max of directive vs your download_delay (only slows down).

Checkpoint resume — crawldir stores pending requests + visited URLs; Ctrl+C saves, restart resumes.

Mixed sessions per spider — public pages use fast FetcherSession; protected pages use lazy AsyncStealthySession. Example shows configure_sessions registering both, then Request(link, sid="stealth") vs sid="fast".

Built-in spider templates — CrawlSpider (rule-based), SitemapSpider, XMLFeedSpider / CSVFeedSpider (RSS/CSV), ShopifySpider (one variant per row), ContentSpider (full-site to Markdown corpus).

Dev-mode cache — first run caches responses to disk; replays for parser iteration without hitting target.

Proxy rotation — ProxyRotator with round-robin/custom strategies, covers all session types, per-request override; mutually exclusive with proxy param.

Export — result.items.to_json(), to_jsonl(), to_csv(), to_xml().

Beyond Python: CLI, MCP Server, Agent Skills, RAG

CLI extraction — scrapling extract get 'url' out.md, --css-selector, --impersonate, --wait-selector for JS pages. Output: .md (Markdown), .txt (text), .html.

MCP Server (16 tools) — half one-shot ( make_request, browser_fetch_once), half sessionful ( browser_open, browser_fetch, browser_actions, browser_screenshot …). browser_actions chains clicks, form fills, hotkeys, alert handling, waits in one call — avoids round-trips per action. browser_screenshot returns actual image blocks. Auto-sanitizes prompt-injection vectors: hidden elements, aria-hidden, zero-width chars, HTML comments, <template> tags.

Agent Skill — Markdown docs per AgentSkills spec so coding assistants (Claude Code, OpenClaw) "learn" current API instead of hallucinating from stale training data. Install: npx skills add D4Vinci/Scrapling --skill scrapling-official.

RAG-friendly — page.markdown() strips scripts/styles/hidden content for clean corpus ingestion.

Cold Water: Limitations & Gotchas

Library, not platform — no visual scheduler, cloud cluster, result DB, or API. Operational tooling (Scrapyd, Crawlab) must be added separately.

Browser tier costs real money — each StealthyFetcher spawns Chromium; memory tier 3★ vs HTTP tier 1★. disable_resources gives 25% speedup; implied baseline is 25% slower. Large-scale runs need budget planning.

Cloudflare solver has boundaries — custom implementations may need manual wait_selector; solver timeout ≥60s; toughest sites may resist generic solver.

Fingerprint ≠ universal pass — Issue #472: Docker image, residential IP, no proxy, targeting Imperva-protected site. All four modes (curl_cffi Chrome, Playwright, stealth, stealth+Cloudflare) blocked, returning 83-byte "Request unsuccessful". TLS+stealth is strong but not a master key; other rule sets may block regardless of Chrome-likeness.

Zero-dep install trap — pip install scrapling only installs parser. Fetchers need pip install scrapling[fetchers] + scrapling install (downloads browser/sys deps). Direct import scrapling.fetchers raises ModuleNotFoundError.

v0.5 is breaking — release notes warn of too many breaking changes; migration guide required. Issues report upgrade breakage.

Not built for massive scale — author says "hundreds of developers daily use for past year" — personal/small-team scale. Millions of URLs need external task queue + distributed scheduler.

Thin community knowledge base — no Scrapy-level tutorials. Best third-party review: The Web Scraping Club Nov 2025 hands-on, but tested v0.3.7 (pre-crawler-engine, pre-AutoThrottle, pre-proxy-rotation). Chinese tutorials nearly absent. You'll likely read source or hit Discord for answers.

Adaptive only works if content stable — if copy rewrites, similarity match fails. Fundamental, not implementation flaw.

Benchmarks are self-run — 1.99ms vs 3412ms on author's script, 100-run average. Not predictive for your page structures.

Return types inherit str — early versions leaked wrapper objects (e.g., TextHandlers object has no attribute partition, issue #41). v0.4+ stabilized, but type-sensitive downstream libs may still choke.

Author's Verdict

Scrapling bets that redesigns change shells, not content. In practice, this holds for most cases — e-commerce layouts shift, class names flip from pinyin to English, but "this button adds to cart" stays true. Adaptive rescues most redesign-killed scrapers (most, not all). If button text rewrites, manual maintenance returns.

Real value: merges Scrapy's mature engineering (concurrency, checkpoint, robots, throttle) with a modern parser, plus a unique AI-agent path (MCP + Skill). 95k stars earned.

Decision heuristic: one-off vs long-running . One-off: requests + BeautifulSoup in 10 minutes; Scrapling is overkill. Long-running, dozens/hundreds of sites, weekly redesign battles — Scrapling's saved maintenance time likely outweighs learning curve and dependency weight.

One-liner: It downgrades "site redesign" from catastrophe to minor patch, but doesn't eliminate it. For persistent data pipelines, that downgrade alone pays off.

Open source
github.com/D4Vinci/Scrapling
scrapling.readthedocs.io