How We Built a Fully Automated Cloud‑Based Web Recording Service for Douyu Ads

Background

Customers place advertisements on Douyu and need screen recordings (video plus danmu) to review ad performance and interaction metrics. Previously, account executives manually recorded using OBS, uploaded files to a network drive, and clients had to download them, which was cumbersome, time‑consuming, and error‑prone.

To eliminate this manual workflow, an automated solution was required to free staff from repetitive tasks.

Through close collaboration with the technical support department, FED, and WSD, a cloud‑based web recording service was built that records entirely without human intervention, uploads the result to Douyu OSS, and provides a direct link for customers to view.

Technology Stack

The business requires not only the video stream but also real‑time danmu, gift rendering, and key ad components, making server‑side composition infeasible. Therefore, client‑side browser automation is used to capture the page content.

Multiple technologies spanning front‑end, back‑end, database, operating system, and cloud are employed:

Nodejs – implements the recording workflow and provides HTTP services

Puppeteer – controls the browser for automation

Chrome DevTools Protocol – offers lower‑level automation capabilities

Chrome Extension – customizes the page during recording

Chrome.tabCapture API – captures the tab content

Ubuntu – base operating system for the recording program

xvfb – provides a virtual desktop on headless Linux servers

x11vnc – enables remote VNC access for debugging

ffmpeg – transcodes the recorded WebM to MP4/AAC

Docker – packages the entire environment

MongoDB – stores recording metadata

Node Express – implements the HTTP API

ack‑keda – triggers recording pod instances based on task queues

Platforms used include Tiano (Douyu internal build and release system), a multi‑cloud database management system, and Alibaba Cloud Log Service (SLS) for log querying.

Overall Process

keda

launches a corresponding number of recording pod instances according to the task queue; these pods may be distributed across multiple cloud hosts, achieving concurrent multi‑stream recording.

When recording finishes, the main process exits, and the pod resources are released, providing elastic resource usage.

Details

Chrome Extension

The extension provides page customization and enhancement capabilities. It consists of three main parts:

manifest.json – describes the extension (name, version, permissions, etc.)

content‑script – injected script that can modify the DOM while sharing the page context

background – runs privileged code and communicates with content‑scripts

Two example features implemented:

Displaying Beijing Time

A div is injected into the page, updated every second, and styled via CSS to appear at the bottom‑right corner.

// content‑script.js
const div = document.createElement("div");
div.id = "tab-recorder-timer";
div.innerText = getTimerText();
div.dataset.filename = "recorded.webm";
setInterval(() => { div.innerText = getTimerText(); }, 1000);
document.body.append(div);

Time zone conversion is handled with the Intl.DateTimeFormat API:

const formatter = new Intl.DateTimeFormat("zh-cn", {
  year: "numeric",
  month: "numeric",
  day: "numeric",
  hour: "numeric",
  minute: "numeric",
  second: "numeric",
  timeZone: "Asia/Shanghai",
  hourCycle: "h23"
});

Tab Capture with chrome.tabCapture

The chrome.tabCapture API can only be called from the background script. Because Chrome requires a user gesture to start capture, the injected time‑display div is clicked to trigger the capture, and a message is sent to the background script.

// background.js
chrome.tabCapture.capture({audio:true, video:true}, stream => {
  const recorder = new MediaRecorder(stream);
  recorder.start();
  // handle dataavailable, stop, etc.
});

To avoid the permission dialog that normally appears, the extension ID is added to Chrome's whitelist via the launch argument --whitelisted-extension-id=${Config.EXTENSION_ID}. The extension ID is made deterministic by signing the extension; a script for generating the signing key and ID is provided.

# Create private key
openssl genrsa 2048 | openssl pkcs8 -topk8 -nocrypt -out key.pem
# Generate public key string for manifest.json
openssl rsa -in key.pem -pubout -outform DER | openssl base64 -A
# Calculate extension ID
openssl rsa -in key.pem -pubout -outform DER | shasum -a 256 | head -c32 | tr 0-9a-f a-p

Automated Recording

Puppeteer (or puppeteer‑core with a custom Chrome binary) drives the browser. Headless mode is disabled because the Chrome extension must be loaded.

import puppeteer from "puppeteer-core";
const browser = await puppeteer.launch({
  headless: false,
  executablePath: Config.CHROME_PATH,
  ignoreDefaultArgs: true,
  args: [
    "--disable-background-networking",
    "--enable-features=NetworkService,NetworkServiceInProcess",
    "--disable-background-timer-throttling",
    "--disable-backgrounding-occluded-windows",
    "--disable-breakpad",
    "--disable-client-side-phishing-detection",
    "--disable-component-extensions-with-background-pages",
    "--disable-default-apps",
    "--disable-dev-shm-usage",
    "--disable-features=Translate",
    "--disable-hang-monitor",
    "--disable-ipc-flooding-protection",
    "--disable-popup-blocking",
    "--disable-prompt-on-repost",
    "--disable-renderer-backgrounding",
    "--disable-sync",
    "--force-color-profile=srgb",
    "--metrics-recording-only",
    "--password-store=basic",
    "--use-mock-keychain",
    "--enable-blink-features=IdleDetection",
    "--autoplay-policy=no-user-gesture-required",
    `--load-extension=${Config.EXTENSION_PATH}`,
    `--disable-extensions-except=${Config.EXTENSION_PATH}`,
    `--whitelisted-extension-id=${Config.EXTENSION_ID}`,
    "--no-first-run",
    "--no-default-browser-check",
    "--no-sandbox",
    "--start-maximized"
  ],
  defaultViewport: {width:1440, height:900}
});

Page navigation and interaction:

const page = await browser.newPage();
await page.goto(task.url);
// Click the injected timer div to start recording
await page.$eval("#tab-recorder-timer", (ele, filename) => {
  ele.dataset.filename = filename;
  ele.click();
}, `${task.filename}.webm`);
// Click again to stop and trigger download
await page.click("#tab-recorder-timer");

Download handling is performed via Chrome DevTools Protocol to set a custom download directory and listen for completion:

const client = await browser.target().createCDPSession();
await client.send("Browser.setDownloadBehavior", {
  behavior: "allow",
  downloadPath: Config.DOWNLOAD_PATH,
  eventsEnabled: true
});
client.on("Browser.downloadProgress", progress => {
  if (progress?.state === "completed") {
    resolve();
  }
});

Transcoding

The recorded WebM (h264+opus) is transcoded to MP4 (h264+aac) using ffmpeg for broader compatibility:

import { exec } from "child_process";
import { promisify } from "util";
await promisify(exec)(`ffmpeg -i "${input}" -y -acodec aac -vcodec copy "${output}"`);

Page Rule System

A generic rule engine allows per‑site customizations such as disabling P2P, forcing low resolution, hiding pop‑ups, repositioning the player, and request whitelisting/blacklisting. Rules are defined as TypeScript interfaces and applied before/after page load.

export interface Rule {
  /** URL match regex */
  match: RegExp;
  /** Network request whitelist */
  networkWhiteList?: RegExp[];
  /** Network request blacklist */
  networkBlackList?: RegExp[];
  /** Hook before page load */
  onBefore?: (page: Page) => Promise<void>;
  /** Hook after page load */
  onAfter?: (page: Page) => Promise<void>;
}

Uploading

After transcoding, the file is uploaded via Douyu's internal WSD upload service, the URL is stored in MongoDB, and the task status is marked as successful for downstream queries.

Database

MongoDB stores recording tasks. Mongoose defines the schema and provides CRUD utilities.

import mongoose from "mongoose";
const RecordTaskSchema = new mongoose.Schema({
  url: String,
  filename: String,
  recordTime: Number,
  createTime: {type: Date, default: () => new Date()},
  status: {type: String, index: true},
  filepath: String,
  from: String,
  recorderName: String,
  ver: String
});
const RecordTaskModel = mongoose.model("recordtask", RecordTaskSchema);

HTTP Service

A minimal Node Express service exposes two endpoints: /record to create a task and /query/:taskId to retrieve task status and result URL.

import express from "express";
const app = express();
app.use(express.json());
app.post("/record", async (req, res) => {
  const {url, filename, recordTime, from} = req.body;
  const task = new RecordTaskModel({url, filename, recordTime, from, status: "waiting"});
  await task.save();
  res.send({code:0, msg:"ok", taskId: task.id});
});
app.get("/query/:taskId", async (req, res) => {
  const task = await RecordTaskModel.findById(req.params.taskId).exec();
  if (!task) return res.send({code:-1, msg:`taskId=${req.params.taskId} don't exist.`});
  res.send({code:0, msg:"ok", data:{id:task.id, url:task.url, filename:task.filename, recordTime:task.recordTime, status:task.status, createTime:task.createTime.getTime(), filepath:task.filepath, recorderName:task.recorderName}});
});
app.listen(80);

Docker Packaging

The entire environment is containerized with a Dockerfile based on Ubuntu 18.04. It installs system packages (xvfb, ffmpeg, x11vnc, fonts), Node.js 14, Chrome, copies source code, runs npm install, builds, and produces a lightweight production image.

FROM ubuntu:18.04
ARG TZ="Asia/Shanghai"
ARG WORKSPACE="/workspace"
WORKDIR ${WORKSPACE}
ENV DEBIAN_FRONTEND=noninteractive
ENV NODE_ENV=production
COPY resource /resource
RUN set -e \
  && sed -i s@/archive.ubuntu.com/@/mirrors.163.com/@g /etc/apt/sources.list \
  && apt-get update \
  && apt-get install -y xvfb tzdata curl fonts-noto-cjk fonts-noto-color-emoji gnupg ffmpeg x11vnc \
  && ln -sf /usr/share/zoneinfo/${TZ} /etc/localtime \
  && echo ${TZ} > /etc/timezone \
  && dpkg-reconfigure -f noninteractive tzdata \
  && apt-get install /resource/nodejs/nodejs_14.17.2-1nodesource1_amd64.deb -yq \
  && apt-get install /resource/chrome/google-chrome-stable_current_amd64.deb -yq \
  && rm -rf /var/lib/apt/lists/* /resource
COPY chrome-extension chrome-extension
COPY scripts scripts
COPY src src
COPY package.json package.json
COPY package-lock.json package-lock.json
COPY tsconfig.json tsconfig.json
RUN npm install --also=dev --registry=https://registry.npm.taobao.org \
  && npm run build \
  && rm -rf node_modules \
  && npm install --production --registry=https://registry.npm.taobao.org

Running Containers

Images are launched with docker run for the server and recorder, or orchestrated with docker‑compose for simplified multi‑service deployment.

# docker run -p 80:80 dy_recorder:1.0 scripts/start_server.sh
# docker run -p 5920:5920 dy_recorder:1.0 scripts/start.sh
# docker-compose.yml example
version: "3"
services:
  server:
    image: dy_recorder:1.0
    ports: ["80:80"]
    entrypoint: "scripts/start_server.sh"
  recorder:
    image: dy_recorder:1.0
    ports: ["5920:5920"]
    entrypoint: "scripts/start.sh"

The solution has been deployed to production, recording over 500 videos with 100% success rate, supporting up to 30 concurrent recordings, and demonstrates high availability and reliability.