Tired of Writing Crawlers? Build a Cross‑Platform Hotspot Dashboard with a Single API

I previously spent weeks writing Python scripts with requests to scrape hot content from various social platforms, only to face captchas, IP blacklists, and endless proxy and header tweaks that made the code increasingly messy.

Maintaining those crawlers eventually cost as much time as manually curating the data, and every platform change required another code update.

One day I discovered a GitHub project that used RedFox's API to aggregate data from Douyin, Xiaohongshu, WeChat public accounts, and Bilibili. The project simply called the API, scheduled a nightly job, and fed the results to an LLM for analysis.

While the idea was appealing, I needed a more focused solution that tracked only the "AI programming tools" niche across selected accounts, and I wanted an interactive web panel rather than a static page.

Technology Stack

The final tool uses pure HTML for the front end, a Python Flask back end, and stores data locally in sqlite3. All data comes from RedFox's API.

Project Structure

redfox_skill_dashboard/
  app.py               # Python back end
  tracker_config.json  # Account configuration
  tracker.db           # SQLite database
  .env                 # RedFox API key
  static/
    index.html        # Page layout
    app.js            # Front‑end interaction
    styles.css        # Page styling

Back‑End Design

The back end performs four tasks: read the API key, save account configuration, call RedFox API to fetch data, and store the results in SQLite for the front end.

def call_api_url(method: str, url: str, payload: dict) -> dict:
    key = api_key()
    headers = {
        "Accept": "application/json",
        "Content-Type": "application/json",
        "X-API-KEY": key,
        "X-API-Key": key,
        "User-Agent": "RedFoxMonitor/1.0",
    }
    body = json.dumps(payload, ensure_ascii=False).encode("utf-8")
    req = urllib.request.Request(url, data=body, headers=headers, method=method)
    with urllib.request.urlopen(req, timeout=30) as resp:
        return json.loads(resp.read().decode("utf-8"))

Example for fetching Xiaohongshu data:

raw = call_api_url(
    "POST",
    "https://redfox.hk/story/api/xhsUser/query",
    {
        "userIds": ids,
        "source": "AI编程工具账号监控-小红书",
    },
)

After receiving the account data, the script extracts the list of works with:

works = acc_row.get("works") or acc_row.get("workList") or acc_row.get("notes") or []

Targeted Monitoring

I monitor three categories for the "AI programming tools" niche:

WeChat public accounts: 10 target accounts, fetch latest articles daily.

Xiaohongshu: 20 relevant accounts, track rising notes.

Douyin: 10 top accounts, track video metrics.

Final Dashboard

The web panel shows the latest hot articles, notes, and videos with a single click; results appear within ten seconds.

Additional tabs display Douyin hot AI videos and Xiaohongshu trending notes.

Running Locally vs. Server

The dashboard runs locally on Flask at localhost:8765. If remote access is needed, it can be deployed to a cheap cloud VM with Gunicorn in about ten minutes. No GitHub Actions or scheduled jobs are required; the user refreshes on demand.

Cost Analysis

RedFox charges per call (≈ 0.02 CNY). Monitoring 40 accounts with 5 items each yields about 200 calls daily; adding historical comparisons brings it to ~300 calls. At 0.02 CNY per call, the daily cost is ~6 CNY, under 200 CNY per month—significantly cheaper than the previous proxy and server expenses.

The API also handles failures gracefully: a single platform outage does not affect others, time‑outs fall back to alternative skills, and inter‑call delays of 0.15–0.25 s prevent rate‑limit bans.

Conclusion

RedFox's popular skills (e.g., 10 k+ article recommendation, Douyin hot account recommendation) see tens of thousands of calls, indicating stability. Using the dashboard, I now spend five minutes each morning reviewing three tabs, selecting promising topics, and drafting articles three times faster than before.

Overall, the API‑driven approach eliminated messy crawler code, reduced operational costs, and provided a clean, maintainable data panel.

https://redfox.hk/

https://github.com/redfox-data/redfox-community/tree/main/skills