Guide to Capturing and Parsing Douyin User Data with Fiddler, mitmproxy, and Python

Preparation: install Fiddler (v5.0.20202.18177), mitmproxy, Nox emulator (7.0.0.6), Python 3.6 with PyCharm, and ensure a CSV file can be used for data storage.

Overall workflow: use Fiddler to capture Douyin HTTP traffic, configure the emulator to route through the proxy, and enable automatic JSON saving via a custom script.

Fiddler script (C#) intercepts responses, extracts the JSON body, and writes it to a file with a unique name.

static function OnBeforeResponse(oSession: Session) {
    if (m_Hide304s && oSession.responseCode == 304) {
        oSession["ui-hide"] = true;
    }
    // 1. Save user info script
    if (oSession.uriContains("https://aweme-eagle.snssdk.com/aweme/v1/user")) {
        var strBody = oSession.GetResponseBodyAsString();
        var sps = oSession.PathAndQuery.slice(-58,);
        var filename = 路径 + "/" + sps + ".json";
        var curDate = new Date();
        var sw : System.IO.StreamWriter;
        if (System.IO.File.Exists(filename)) {
            sw = System.IO.File.AppendText(filename);
            sw.Write(strBody);
        } else {
            sw = System.IO.File.CreateText(filename);
            sw.Write(strBody);
        }
        sw.Close();
        sw.Dispose();
    }
}

Alternatively, use mitmproxy with a Python script to capture responses and save each JSON payload to a uniquely named file.

import os
import mitmproxy.http
from mitmproxy import ctx

def response(flow: mitmproxy.http.HTTPFlow):
    path = 'E:/dy_project/dy_data_mitmdump'
    try:
        os.mkdir(path)
    except:
        pass
    aim_url = 'https://aweme-eagle.snssdk.com/aweme/v1/user'
    if flow.request.url.startswith(aim_url):
        json_data = flow.response.content
        filename = 'dy_u_data' + flow.request.url.split('&ts=')[-1].split('&as=')[0]
        with open(path + '/' + filename + '.json', 'wb') as f:
            f.write(json_data)
        ctx.log.info(f'**********{filename}.json下载了**********')

Run mitmdump with the script, optionally using a .bat file for convenience, e.g., mitmdump -q -s save_data_to_json.py , and adjust options such as body size limits or request method filters.

Parse the saved JSON files with Python, extracting fields like nickname, uid, gender, country, city, province, school name, birthday, verification status, video count, likes, followers, and fans, then write the results to a CSV file.

import os
import json

def parse(self, _json):
    file = open(_json, 'r', encoding='utf8')
    for line in file.readlines():
        json_data = json.loads(line)
        user = json_data.get('user')
        user_name = user.get('nickname')
        user_uid = user.get('uid')
        user_id = user.get('unique_id')
        user_gender = user.get('gender')
        user_country = user.get('country')
        user_city = user.get('city')
        user_province = user.get('province')
        user_school_name = user.get('school_name')
        user_birthday = user.get('birthday')
        user_verify = user.get('custom_verify')
        user_aweme_count = user.get('aweme_count')
        user_favorite = user.get('favoriting_count')
        user_signature = user.get('signature')
        user_focus = user.get('following_count')
        user_appreciate = user.get('total_favorited')
        user_fans = user.get('mplatform_followers_count')
        # Save extracted data as needed

The guide also mentions using Nox's recording feature for automated video scrolling and shows example CSV output of the collected user data.