Free Viewpoint Video Technology Optimization at Youku

Youku’s free‑viewpoint video mode offers an immersive, multi‑angle viewing experience that has attracted significant user attention. However, early deployments suffered from playback stutter, unclear images, and limited device coverage.

In the first half of the year, the Youku technical team performed a comprehensive upgrade of the free‑viewpoint feature. The following sections describe the overall goals, the client‑side architecture, performance‑optimization strategies, and the resulting impact.

1. What is Free Viewpoint? Free‑viewpoint video adds a depth‑map (per‑frame depth image) to the normal video stream. An SDK processes each depth map to synthesize frames from arbitrary angles, which are then displayed to the user.

2. Client Architecture Design

The architecture consists of four layers:

Playback Business Layer: Handles user interactions such as angle‑rotation controls, UI prompts, and transition animations; features a switch to enable/disable the function online.

Player Middleware Layer: Modifies the playback pipeline to support free viewpoint and downloads the required algorithm files, passing their paths to the algorithm layer.

Player Core Layer: Exchanges data between the core and the algorithm SDK, composites the processed texture, and renders it on screen.

Downloader: Uses multi‑segment downloading to fetch video and algorithm files efficiently.

Algorithm Layer: Reconstructs the target‑angle view from the depth map.

3. Performance Optimization方案

The team investigated the causes of stutter and identified three main levers: pre‑fetching data, multi‑channel downloading, and lowering video bitrate. After feasibility studies, four solutions were selected:

Pre‑caching

Intelligent bitrate adaptation (smart‑segment)

Dynamic kernel buffer sizing

Multi‑channel downloading

4. Optimization Practices

Smart‑segment : An algorithm decides the bitrate of the next TS segment based on buffer length, network conditions, and playback state, allowing dynamic bitrate reduction.

Dynamic Kernel Buffer : Configuration‑driven adjustment of the kernel buffer size maximizes download resource utilization.

Multi‑Channel Download : Files are split into N chunks, each downloaded over a separate channel in parallel, greatly improving download speed and reducing stutter.

Pre‑Cache : Caches playback control information and video streams; supports dynamic cache size configuration; unifies free‑viewpoint configuration in the backend to avoid client‑backend mismatches.

5. Data Comparison & Effect

After optimization, the free‑viewpoint version of “Street Dance 4” achieved nearly double the total playback volume compared with the previous season and improved smoothness by about 70%.

Device coverage expanded from roughly 50% (mid‑high‑end devices only) to about 80%, thanks to a downgrade rendering mode that enables low‑end devices with sufficient 4K decoding capability to use free viewpoint.

6. Conclusion

Youku will continue to enhance the free‑viewpoint experience, exploring live‑stream free viewpoint and further technical innovations to provide richer, higher‑quality viewing for users.