AI-Based Follow-Subtitle (Bullet) System for Video Streaming

This article introduces a new interactive subtitle (bullet) feature for video streaming platforms, where subtitles follow characters on screen using AI-powered face recognition. Unlike traditional top‑down scrolling bullets, the follow‑bullet appears as a speech‑bubble attached to a character’s face and moves with the character.

The system architecture consists of three layers: algorithm side, server side, and client side. The algorithm side extracts video frames at 25 fps, performs face detection, tracking, and smoothing to generate per‑frame face metadata. The server aggregates this metadata, applies denoising, anti‑shake processing, and merges frame‑level data into continuous face trajectories, then packages these trajectories together with the corresponding bullet data.

On the client, an interactive SDK loads scripts that represent small interactions. Each script contains a face trajectory and its associated bullet bubble data. A timer polls the current playback time, retrieves the relevant face coordinates, and renders the bullet bubble next to the moving face, achieving a seamless follow‑bullet effect.

The article also explains why face recognition is performed on the server rather than the client: real‑time constraints, insufficient accuracy on mobile devices, and excessive CPU/power consumption would degrade user experience.

Key challenges discussed include handling bullets that appear just before a scene cut (preventing awkward lingering after the cut) and synchronizing face data with video edits that cause timeline misalignment. Solutions involve applying fade‑out effects for cut‑scene bullets and calculating precise cut durations using short segments of face data.

Future work aims to extend the basic script framework beyond follow‑bullets to other interactive features such as bullet‑through‑people, leveraging the same face and body data. This will enable developers to create custom scripts by inheriting the optimized face data pipeline.