Scrape Douyin Hot List on Android with Charles, Appium, and mitmproxy

This article shares a simple method to analyze and capture data from the Douyin (TikTok) hot list on Android.

Since the data is not available on a web page, we need to intercept the app's network traffic using Charles proxy.

Set up Charles

Download and install Charles (a paid proxy tool; a Chinese‑localized version is available). After installation, open Charles and configure the proxy.

On the phone, set the proxy to the computer's IP address and port 8888, ensuring both devices are on the same LAN.

Install certificates

Install the Charles root certificate on both the PC and the mobile device (Help → SSL Proxy → Install Charles Root Certificate). On the phone, open chls.pro/ssl to download the certificate, then go to Settings → Security → Install from SD card.

Capture Douyin hot list data

Open Charles, launch the Douyin app, and navigate to the hot list page. The request URL appears in Charles and returns 50 items.

The endpoint only returns data for the current moment; obtaining newer data requires changing complex parameters, which is beyond the scope of this guide.

Automate data collection

Use Appium to periodically control the phone and mitmproxy to intercept the traffic. The Appium script repeatedly fetches the latest hot list, while the mitmproxy script filters requests whose URL matches the hot list endpoint.

Before using mitmproxy, set the phone's proxy IP and port to 127.0.0.1:8080.

Include the first two lines in the script to avoid runtime errors.

Use if url in flow.request.url to identify Douyin hot list requests.

Run the script from its directory.

Running the Appium automation completes the data collection process.