How SkyReels V4 Achieves Synchronized Audio‑Video Generation at Film Quality

Overview

SkyReels V4 is a multimodal diffusion model that generates ultra‑high‑definition video sequences up to 15 seconds long with perfectly synchronized audio.

Dual‑Stream Architecture

The backbone consists of two symmetric branches built on a Multi‑Modal Diffusion Transformer (MMDiT). The video branch is initialized from a pretrained video model, while the audio branch is trained from scratch. Early layers keep separate parameter spaces for video and audio; later layers merge via cross‑attention, allowing each modality to attend to the other and preserving tight sync. Rotational Position Encoding (RoPE) scales audio frequencies to match the coarser video timeline, aligning the two temporal resolutions.

Unified Channel Concatenation

Input to the video generator is formed by concatenating three tensors along the channel dimension:

Noisy latent video variables.

Reference frames (or video clips) encoded by a variational auto‑encoder (VAE) and resized to a uniform pixel size.

A binary mask indicating which regions should be generated (0) or preserved (1).

A mask of all zeros triggers pure text‑to‑video generation; a mask with the first frame set to one enables image‑to‑video generation; arbitrary spatial‑temporal masks allow precise editing. Reference embeddings are shifted with three‑dimensional RoPE so that they occupy distinct temporal positions from the target generation timeline.

Step‑wise Refinement for Long High‑Resolution Sequences

The model first produces a low‑resolution long sequence together with high‑resolution keyframes. A dedicated Refiner module, inheriting weights from a pretrained video generator, upsamples the low‑resolution frames and performs frame‑rate interpolation, replacing interpolated frames at keyframe positions with the high‑resolution predictions.

To mitigate the quadratic cost of attention on long sequences, SkyReels V4 adopts Video Sparse Attention (VSA). Tokens are pooled to identify salient spatio‑temporal blocks, and dense attention is applied only within the top‑ranked blocks, reducing computation to roughly one‑third of the full cost.

Curriculum Training

The data pipeline aggregates public datasets and licensed film/short‑video content, filters out watermarks and low‑quality samples, and enforces strict audio‑video sync filtering. Training proceeds in stages:

Low‑resolution text‑to‑image pre‑training to learn semantic grounding.

Video pre‑training to capture motion dynamics and temporal coherence.

Joint audio‑video training on synchronized tasks: text‑to‑video, text‑to‑audio‑video, and text‑to‑audio.

Fine‑tuning on carefully curated high‑quality data to polish visual fidelity and synchronization precision.

SkyReels‑VABench Evaluation

The authors introduce SkyReels‑VABench, a benchmark covering instruction compliance, audio‑video sync, visual quality, motion quality, and audio quality. In blind tests with 50 professional evaluators, SkyReels V4 ranked second globally in sync generation, outperforming models such as Veo 3.1, Vidu Q3, Sora 2, and Wan 2.6, and achieved top scores in instruction compliance and motion quality.

Reference: https://arxiv.org/pdf/2602.21818