AI-Powered Video Intelligence: Architecture, Optimization, and Application Scenarios

The June 29 audio‑video and unified communications technology salon concluded with a technical deep‑dive by Tencent Cloud experts on low‑latency streaming, commercial live‑broadcast solutions, and especially the application of AI in video intelligence.

The presentation was divided into three parts. The first part introduced three "video + AI" products: Ultra‑Fast HD , which reduces bitrate without sacrificing quality by recognizing video scenes and applying scene‑specific encoding parameters; Cloud Clip , a web‑based online video editing tool; and Zhimo , a comprehensive video‑AI platform offering intelligent recognition, editing, and review.

Zhimo’s intelligent recognition can identify persons, convert speech to text, perform OCR on on‑screen text, and detect objects such as logos. Its editing capabilities include automatic categorization, tagging, cover generation, and compilation of highlights. The review module can filter prohibited content (e.g., pornography, political, violent, or illegal subtitles) and perform voice‑activity detection.

The system architecture of Zhimo consists of four layers: access, logical processing, model recognition, and data storage. The workflow starts with user‑managed face libraries and sensitive‑word lists, proceeds to verification, then launches video processing tasks that include frame extraction, audio extraction, and multi‑modal recognition. Results are aggregated and returned via configurable policies.

Deployment supports both public‑cloud and private‑cloud (on‑premises) scenarios, allowing customers with sensitive data to keep processing within their own data centers while still leveraging unified authentication, VOD management, and billing services.

Video processing follows a multimedia pipeline: file input (VOD, live, local), demuxing, decoding, frame and PCM extraction, resampling/transcoding for ASR engines, and parallel queues for simultaneous recognition of frames and audio. The system dynamically adjusts queue consumption rates and download speeds to meet various processing speed ratios (e.g., 5‑10× real‑time).

Optimization techniques include:

Scene‑aware face retrieval using vector similarity (TOP‑K and range queries) with three modes: library‑based, historical scan, and no‑library detection.

Context‑aware threshold adjustment based on ASR or OCR cues (e.g., lowering face‑similarity thresholds when a name appears in subtitles).

Seamless engine upgrades via multi‑version data layers and hot‑restart switching.

Voice Activity Detection (VAD) to discard silent audio segments before ASR, reducing bandwidth and improving accuracy.

Efficient frame filtering using simple histogram statistics instead of heavy algorithms (PSNR, perceptual hash, SIFT, CNN).

Region‑focused OCR processing to speed up text extraction.

Content‑driven video summarization: key‑frame detection, anchor‑person detection, and automatic clipping of news segments, highlights, and intro/outro removal.

Application scenarios highlighted include media asset management (searching videos by celebrity appearance), video recommendation, live‑stream monitoring for prohibited content, large‑scale video audit (millions of videos per day), real‑time subtitle generation, and automated clip creation for sports highlights or news segments.

A Q&A session revealed interest in extending the platform to behavior detection (e.g., motion speed, acceleration), which is not yet a product but could be developed on demand.