How We Built an Automated Video Quality Monitoring System with Machine Learning

Background

2017 marked a turning point for mobile live streaming, and Huajiao Live distinguished itself not only through high‑quality content but also by delivering a superior video playback experience. The following is a technical summary of Huajiao's video quality monitoring system.

Architecture

Logical Structure

Sampling Layer: Extract key frames from the video stream at regular intervals (2–4 s) to represent short video segments.

Video consists of consecutive frames; extracting key frames provides a universal analysis method.

Analysis Layer: Apply a pre‑trained machine‑learning model to each key frame to obtain a classification result, which serves as an automatic score.

The model processes each image and outputs a quality classification.

Business Processing Layer: Aggregate sliding‑window results from recent frames to smooth out prediction errors and produce a stable video quality score.

A cache window stores recent analysis results; each new result triggers a recomputation of the overall quality score.

Interface/Presentation Layer: Expose the quality assessment via API for downstream services.

Current Deployment Physical Topology

The system builds on Huajiao's existing real‑time screenshot service and business APIs.

Video analysis servers retrieve live stream metadata via business APIs.

Load balancers distribute monitoring tasks to worker machines.

Workers fetch key‑frame screenshots, preprocess them into feature vectors, and run predictions with the trained model.

Results are stored and made available through web dashboards, subscription APIs, email and app alerts.

System Load Analysis

During peak periods, the system processes tens of thousands of streams, updating a screenshot every 2 s. Each image is ~150 KB, requiring dozens of gigabits of bandwidth. With 32‑core servers handling ~20 images per second per core, dozens of machines are needed to meet real‑time demands.

Data Flow Explanation

Periodically collect global stream status as JSON dictionaries and store them in Redis, sharding by MD5 hash prefixes.

Image preprocessing modules poll Redis for assigned stream IDs, download corresponding screenshots, analyze them, and write results to a shared "svm_result" table.

Quality analysis modules aggregate cached results within a sliding window, generate final scores, and output HTML/JSON reports.

External subscription servers consume the JSON reports to trigger email or push alerts.

Outlook

The current system uses a shallow SVM model, which works with limited training data but struggles with some scenarios. Future work will explore deep learning frameworks such as Caffe, TensorFlow, or PyTorch to improve accuracy by leveraging larger labeled datasets.

Summary

Video Quality Evaluation Methods

Objective Evaluation : Compare the original video with processed versions using metrics like PSNR, MSE, SSIM, and gradient‑based sharpness indicators (Brenner, Tenengrad).

Subjective Evaluation : Human raters score video quality according to predefined criteria.

Machine Learning Notes

SVM (Support Vector Machine) : A shallow classifier that can be extended with kernel functions to handle non‑linear separations. In this system, SVM classifies image quality based on extracted gradient features.

Deep Learning (CNN) : Convolutional Neural Networks extract hierarchical features through convolution, pooling, and fully‑connected layers, achieving higher accuracy when trained on abundant data.