How to Automate Douyin Video Scraping with Python, mitmproxy & Appium

Overview

This article demonstrates how to build an automated Douyin video scraper on Windows using Python, mitmproxy for network capture, and Appium for mobile UI automation.

Required Tools

IDE: PyCharm

Network capture: mitmproxy (or its CLI variant mitmdump)

Mobile automation: Appium

Device: Android phone connected via USB

Workflow

Use mitmproxy to intercept the app’s HTTP requests and locate video URL prefixes ( http://v1-dy.ixigua.com/, http://v3-dy.ixigua.com/, http://v9-dy.ixigua.com/).

Run a mitmdump script that filters these URLs and saves the video files.

Employ Appium to automate scrolling and clicking in the Douyin app so that new video requests are continuously generated.

mitmproxy Capture

After installing mitmproxy and configuring the CA certificate on the phone, run mitmdump while the Douyin app is open. The console will display all requests, and the video URLs appear with the three prefixes mentioned above.

Extract the URLs and feed them to a Python script:

import requests
path = 'D:/video/'
num = 1788

def response(flow):
    global num
    target_urls = ['http://v1-dy.ixigua.com/', 'http://v9-dy.ixigua.com/', 'http://v3-dy.ixigua.com/']
    for url in target_urls:
        if flow.request.url.startswith(url):
            filename = path + str(num) + '.mp4'
            res = requests.get(flow.request.url, stream=True)
            with open(filename, 'ab') as f:
                f.write(res.content)
                f.flush()
                print(filename + '下载完成')
            num += 1

The script saves each video as .mp4 files in the specified directory.

Appium Automation

Configure Appium with the device’s platformName, deviceName, appPackage, and appActivity. The appPackage and appActivity can be obtained from the Android log (search for the Displayed keyword).

{
  "platformName": "Android",
  "deviceName": "Mi_Note_3",
  "appPackage": "com.ss.android.ugc.aweme",
  "appActivity": ".main.MainActivity"
}

Start the Appium server and launch a session to control the Douyin app.

Python script to drive the app:

from appium import webdriver
from time import sleep

class Action():
    def __init__(self):
        self.desired_caps = {
            "platformName": "Android",
            "deviceName": "Mi_Note_3",
            "appPackage": "com.ss.android.ugc.aweme",
            "appActivity": ".main.MainActivity"
        }
        self.server = 'http://localhost:4723/wd/hub'
        self.driver = webdriver.Remote(self.server, self.desired_caps)
        self.start_x = 500
        self.start_y = 1500
        self.distance = 1300

    def comments(self):
        sleep(2)
        self.driver.tap([(500, 1200)], 500)

    def scroll(self):
        while True:
            self.driver.swipe(self.start_x, self.start_y, self.start_x, self.start_y-self.distance)
            sleep(2)

    def main(self):
        self.comments()
        self.scroll()

if __name__ == '__main__':
    action = Action()
    action.main()

Running this script continuously scrolls the feed, causing new video requests that mitmdump captures and downloads.

Result

The combined use of mitmproxy for traffic interception and Appium for UI automation creates a fully automated Douyin video crawler. Occasionally duplicate videos may be downloaded.