After Three Years of Building Scrapers, This One‑Line Tool Beats It

Data acquisition bottleneck

For AI‑driven content and media applications, obtaining up‑to‑date public data from platforms such as Douyin, Xiaohongshu, WeChat public accounts, Bilibili, etc., is extremely difficult because of anti‑scraping measures and frequent interface changes.

Why custom crawlers fail

Writing a crawler initially appears simple, but maintenance costs explode: platform structures change often, parsing rules break, accounts and IP proxy pools must be managed, and each new platform requires a completely new parsing logic. The effort to keep crawlers running outweighs the value for individuals or small teams.

Even using popular open‑source frameworks (Playwright, Puppeteer, Scrapy) only solves the "can we scrape?" problem; they do not guarantee stability or eliminate the need for constant monitoring and fixes.

Limitations of commercial data services

Commercial data products reduce operational hassle but introduce high costs, narrow coverage, and subscription models that are unfriendly to early‑stage experimentation.

RedFox unified public‑data API

RedFox (https://redfox.hk) aggregates public data from more than ten Chinese social platforms into a single REST API. It handles login, signature generation, anti‑scraping, and IP proxying, returning structured fields such as title, author, author ID, publish time, likes, comments, collections, and shares.

Supported platforms include Douyin, Xiaohongshu, Video 号, WeChat public accounts, Kuaishou, Bilibili, Weibo, Toutiao, Zhihu, and others. The WeChat public‑account endpoint provides searchable hot articles, a capability that is traditionally hard to obtain.

Key advantages

One API key unlocks all supported platforms, eliminating per‑platform integration work.

Rich query dimensions and fully structured output simplify downstream storage and model feeding.

Platform changes are handled server‑side; client code remains unchanged.

“Skills” marketplace offers 50+ ready‑made data‑collection and analysis skills that can be plugged into agent platforms such as Coze.

Multi‑platform monitor example (Claude Code)

Example: monitor hot content on Douyin and WeChat public accounts using RedFox’s REST endpoints with a single X-API-KEY header.

import requests
API_KEY = "ak_xxxxxxxx"
BASE = "https://redfox.hk/story/api"

def fetch_gzh_hot(keyword: str, start_date: str, limit: int = 10):
    resp = requests.post(
        f"{BASE}/gzh/search/hotArticle",
        headers={"X-API-KEY": API_KEY},
        json={"keyword": keyword, "startDate": start_date, "endDate": "2026-06-16"},
    )
    arts = resp.json()["data"]["articles"]
    return sorted(arts, key=lambda x: x["totalScore"], reverse=True)[:limit]

Douyin hot list uses a similar endpoint:

def fetch_dy_hot(category: str, date: str, limit: int = 10):
    resp = requests.post(
        f"{BASE}/dy/search/likesRank",
        headers={"X-API-KEY": API_KEY},
        json={"type": category, "startTime": date, "endTime": date},
    )
    return resp.json()["data"]["list"][:limit]

Iterate over predefined industry‑keyword mappings, fetch data from both platforms, and merge results into a unified dashboard dictionary:

INDUSTRIES = [
    {"name": "AI", "gzh_kw": "大模型,AIGC,DeepSeek", "dy_type": "科学普及"},
    {"name": "Finance", "gzh_kw": "财经商业", "dy_type": "财富理财"},
    {"name": "Tech", "gzh_kw": "科技数码", "dy_type": "数码科技"},
]

dashboard = {}
for ind in INDUSTRIES:
    dashboard[ind["name"]] = {
        "wechat": fetch_gzh_hot(ind["gzh_kw"], start_date="2026-06-09"),
        "douyin": fetch_dy_hot(ind["dy_type"], date="2026-06-15"),
    }

The returned fields are already structured (title, author, timestamps, engagement metrics) and can be inserted directly into a database or fed to downstream models without HTML parsing.