How Youku Achieves Real-Time Bullet‑Screen Pass‑Through on Mobile

Background

The bullet‑screen pass‑through feature lets viewers see characters clearly even when the screen is flooded with scrolling comments, greatly improving the viewing experience. Youku introduced this feature and quickly gained user popularity.

Bullet‑Screen Pass‑Through Overview

Two major technical paths exist: a cloud‑side offline human segmentation combined with client‑side rendering, and a fully client‑side real‑time segmentation and rendering. Both require efficient bullet‑screen rendering and pass‑through compositing.

Cloud‑Side Solution

Cloud Processing : AI algorithms generate human segmentation masks offline for each video. The masks are stored as template data in the UPS service. When a client plays the video, it fetches the appropriate mask URL, downloads, decodes, and uses it for pass‑through rendering.

Client Side (MetaPipe & OPR) :

MetaPipe handles mask download, parsing, and decoding, exposing the mask data to the OPR rendering module.

OPR Rendering combines the mask with bullet‑screen textures to produce the final composited frame.

On‑Device Real‑Time Solution

This approach performs human segmentation directly on the mobile device, offering real‑time capability for live streams and lower production cost. The workflow is:

Capture the current video frame while the video is rendering.

Send the frame to the OPR intelligent module, which runs a real‑time segmentation model to produce a mask.

Feed the mask to the OPR rendering pipeline to composite bullet‑screens with the mask, achieving pass‑through.

Key Requirements for Mobile Deployment

Accurate pass‑through without misalignment, flickering, or visual artifacts.

No noticeable increase in CPU/GPU usage or memory consumption that would degrade playback smoothness.

Bullet‑Screen Rendering Technology

Bullet‑screens are rendered using OpenGL/Metal on the GPU for high performance. All rendering on Youku's mobile client is GPU‑accelerated.

Pass‑Through Rendering Pipeline

The pipeline consists of four main steps:

OPR obtains the human contour mask for the current video frame.

The mask is uploaded to the GPU and blurred with a 5‑pixel Gaussian kernel to smooth edges and increase tolerance to mask inaccuracies.

The blurred mask is used as an alpha channel and blended with the bullet‑screen texture, reducing the alpha where the human silhouette appears.

The blended result is presented on screen, creating a three‑layer visual hierarchy: foreground human, middle bullet‑screen, background video.

Engineering Optimizations

Several practical issues were encountered and solved during production:

Edge Artifacts : Applying Gaussian blur to the mask smooths harsh edges around hair and facial features.

Mask‑Video Size Mismatch : Because video aspect ratios differ from full‑screen bullet‑screens, mask coordinates are adjusted to match the scaled video dimensions before blending.

Render‑Target Switching Overhead : To avoid costly pipeline re‑creation when a frame lacks a mask, a 1×1 dummy mask is used, keeping the render target constant and preserving performance.

References

Technical resources consulted include the OpenGL ES documentation, Apple Metal programming guides, and related developer articles.