Automated Douyin Video Scraping with Python, mitmproxy, and Appium

This guide demonstrates how to automatically scrape video URLs from the Douyin (TikTok) Android app using Python, mitmproxy, and Appium.

Tools : Python (PyCharm), mitmproxy/mitmdump for network capture, Appium for mobile automation, Windows 10 environment.

Step 1: Capture network traffic with mitmproxy . Ensure mitmproxy is installed, the phone and PC share the same LAN, and the mitmproxy CA certificate is configured. Run mitmdump while the Douyin app is open; video URLs appear with the prefixes http://v1-dy.ixigua.com/ , http://v3-dy.ixigua.com/ , and http://v9-dy.ixigua.com/ .

Step 2: Python script for mitmdump to download videos . The script filters the captured URLs and saves the video files:

import requests
# file path
path = 'D:/video/'
num = 1788

def response(flow):
    global num
    target_urls = ['http://v1-dy.ixigua.com/', 'http://v9-dy.ixigua.com/', 'http://v3-dy.ixigua.com/']
    for url in target_urls:
        if flow.request.url.startswith(url):
            filename = path + str(num) + '.mp4'
            res = requests.get(flow.request.url, stream=True)
            with open(filename, 'ab') as f:
                f.write(res.content)
                f.flush()
                print(filename + '下载完成')
            num += 1

Execute the script with mitmdump -s scripts.py to save videos as you scroll through the app, though manual scrolling is required at this stage.

Step 3: Automate scrolling with Appium . Install Android SDK, connect the device via USB, enable USB debugging, and configure Desired Capabilities:

{
  "platformName": "Android",
  "deviceName": "Mi_Note_3",
  "appPackage": "com.ss.android.ugc.aweme",
  "appActivity": ".main.MainActivity"
}

Start the Appium server, launch a session, and run the following Python code to control the app and continuously swipe:

from appium import webdriver
from time import sleep

class Action():
    def __init__(self):
        self.desired_caps = {
            "platformName": "Android",
            "deviceName": "Mi_Note_3",
            "appPackage": "com.ss.android.ugc.aweme",
            "appActivity": ".main.MainActivity"
        }
        self.server = 'http://localhost:4723/wd/hub'
        self.driver = webdriver.Remote(self.server, self.desired_caps)
        self.start_x = 500
        self.start_y = 1500
        self.distance = 1300

    def comments(self):
        sleep(2)
        self.driver.tap([(500, 1200)], 500)

    def scroll(self):
        while True:
            self.driver.swipe(self.start_x, self.start_y, self.start_x,
                              self.start_y-self.distance)
            sleep(2)

    def main(self):
        self.comments()
        self.scroll()

if __name__ == '__main__':
    action = Action()
    action.main()

By combining mitmproxy capture with Appium‑driven scrolling, the process becomes fully automated, allowing continuous downloading of Douyin videos without manual interaction.