Transforming Immersive Streaming with Free-Viewpoint Video: Capture to Cloud

Free‑viewpoint video (FVV) is an immersive interactive video technology developed by Youku MoKu Lab that uses 3D reconstruction and rendering to provide six degrees of freedom (6‑DOF) playback, allowing users to rotate, zoom, and move freely while watching live or recorded events.

On‑site Capture

Unlike traditional video, FVV capture requires a synchronized array of dozens to hundreds of cameras connected via Ethernet to a local network. Camera streams are aggregated on a现场 server, pre‑processed, and sent to the cloud for reconstruction. For recorded content, footage is stored on media cards and later uploaded.

Site Survey and Planning

Teams coordinate with production units to assess venue layouts, determine camera placement, and integrate FVV requirements during stage design, ensuring sufficient audio quality and resource allocation for smooth downstream processing.

Hardware System Setup

Before the event, racks, cameras, switches, and routers are installed and synchronized. Camera parameters and poses are calibrated using acquisition software, achieving sub‑second alignment. The modular setup allows rapid re‑configuration, reducing physical build time to about two hours and rehearsal time to under thirty minutes.

Audio‑Video Capture

The 6‑DOF Studio software captures multi‑camera audio‑video streams, monitors system status, and performs real‑time or offline processing, supporting up to 4K input. Live pipelines achieve end‑to‑end latency of roughly five seconds, while supporting 8K streaming and 1080p interactive playback.

Cloud Processing

In the cloud, captured streams undergo multi‑camera calibration, depth estimation, and 3D reconstruction. Calibration solves intrinsic, extrinsic, and distortion parameters using feature‑point matching across cameras. Depth estimation combines deep‑learning models with traditional image processing to produce 270P depth maps in under 20 ms for live streams and 90 s per frame for VOD.

Encoding is optimized by leveraging depth data to adjust video codec parameters, reducing bitrate by ~20 % while maintaining visual quality, and improving smoothness by over 50 %.

Terminal Rendering

Clients use a dedicated FVV SDK (6DOF SDK) to render interactive video. On PC, the FVV editing tool allows key‑frame editing of virtual camera paths; on mobile, the SDK (Android OpenGL/OpenCL, iOS Metal) provides real‑time rendering with support for multiple camera models and path interpolation, achieving sub‑100 ms stream switching.

Free‑Viewpoint Video Editing Tool

The FreeViewVideoEditor runs on Windows, enabling creation of pure‑play, highlight, and bullet‑time videos by editing key frames, previewing results, and uploading to the cloud for final rendering. It supports resolution selection, motion templates, zoom ranges, and audio synchronization, with typical production times of 30 min to 2 h per dance segment.