Self-Forcing: Turning Global Video Diffusion into Causal Streaming for Long-Form Generation

1. From Wan2.1 to Causal Video Diffusion (Self‑Forcing)

Wan2.1 is an open‑source large‑scale video generation model released by Alibaba, built on a diffusion Transformer (DiT) backbone and trained with Flow Matching. It employs Full Spatio‑temporal Attention so that every frame can attend to all other frames, and compresses raw video with a 3D Causal VAE to a 4×8×8 latent grid (latent dimension 16). Text prompts are encoded by umT5 into 512 tokens of 4096‑dimensional embeddings.

Two model sizes are provided: a 1.3 B‑parameter version (1536 hidden size, 30 Transformer layers, 12 attention heads) for resource‑constrained scenarios, and a 14 B‑parameter version (5120 hidden size, 40 layers, 40 heads) for higher quality. During inference the model processes all frames in parallel over 40–50 denoising steps, which works well for short clips but creates memory and latency bottlenecks for longer videos or real‑time use.

2. Bottlenecks of Global Diffusion Models for Long Videos

Because attention complexity scales as O(N²) with token length N, memory usage grows quadratically as video length increases. For a 5‑second, 16 FPS, 832×480 video, the token sequence already reaches ~30 k tokens; doubling the length quadruples memory demand, making single‑GPU generation of long videos impractical.

The model also assumes a fixed maximum frame count during training; at inference time longer videos require sliding‑window or segment‑wise stitching, which introduces visible seams and degrades long‑range consistency.

Moreover, the bidirectional attention prevents streaming inference: each frame depends on future frames, so the system must wait for the entire video to be generated, leading to first‑frame latencies of dozens of seconds—unsuitable for interactive or online generation.

3. Self‑Forcing: Causal Autoregressive Video Diffusion

Self‑Forcing redesigns Wan2.1 into a causal, step‑wise generation model that only accesses historical frames, eliminating the error‑accumulation problem of traditional autoregressive approaches. It introduces a block mask in the attention matrix and uses Flex Attention to keep computation efficient.

During inference, KV caching reuses the Key/Value pairs of already‑generated frames, avoiding redundant attention calculations. The model processes video in chunks (e.g., three latent frames per chunk); after each chunk the results are written to the KV cache, and a rolling cache evicts the oldest tokens when capacity is reached, enabling arbitrary‑length generation within limited GPU memory.

This design reduces the peak attention complexity from O(N²) to O(B×N), where B is the chunk size, and brings first‑frame latency down to sub‑second levels while maintaining generation quality comparable to Wan2.1 (VBench scores slightly improved).

4. Inference Optimization Details

4.1 Sequence Parallelism (SP)

The original Self‑Forcing implementation lacks SP support, which is essential for scaling to longer videos on multi‑GPU setups. By partitioning the sequence dimension across multiple ranks (each rank holds a local slice), the model can distribute memory and compute.

In SP mode, each rank processes its local sequence while applying a causal RoPE that respects global time indices. This avoids the three all‑gather communications required by standard RoPE and enables overlapping communication with computation.

4.2 Causal RoPE Implementation

Wan2.1 uses a 3D Rotary Positional Encoding (RoPE) that splits rotation frequencies into temporal, height, and width components. The original implementation uses complex‑number multiplication:

freqs = freqs.split([c - 2 * (c // 3), c // 3, c // 3], dim=1)

x_i = torch.view_as_complex(x[i, :seq_len].to(torch.float64).reshape(seq_len, n, -1, 2))

freqs_i = torch.cat([
    freqs[0][:f].view(f, 1, 1, -1).expand(f, h, w, -1),
    freqs[1][:h].view(1, h, 1, -1).expand(f, h, w, -1),
    freqs[2][:w].view(1, 1, w, -1).expand(f, h, w, -1)
], dim=-1)

x_i = torch.view_as_real(x_i * freqs_i).flatten(2)

Self‑Forcing adds a start_frame argument to the RoPE routine so that each chunk can compute its own local RoPE based on the global start index:

def causal_rope_apply(x, grid_sizes, freqs, start_frame=0):
    freqs_i = torch.cat([
        freqs[0][start_frame:start_frame + f].view(f, 1, 1, -1).expand(f, h, w, -1),
        freqs[1][:h].view(1, h, 1, -1).expand(f, h, w, -1),
        freqs[2][:w].view(1, 1, w, -1).expand(f, h, w, -1)
    ], dim=-1)

This makes RoPE fully local to each rank, preserving causal consistency without extra cross‑rank communication.

4.3 Performance Gains

By caching sin/cos tables and fusing operators with TileLang, the implementation gains ~10 % over typical Triton kernels. End‑to‑end profiling on a 5 s, 480p clip shows a reduction from 8.86 s to 5.99 s (≈1.48× speed‑up, 47.5 % faster). Additional graph‑level optimizations move dynamic cache logic to pre‑computed tensors stored in contiguous GPU memory, further improving CUDA stream efficiency.

5. Summary and Outlook

The article presents a complete pipeline for converting a full‑frame video diffusion model into a causal, chunk‑wise generator capable of streaming inference, and details the engineering optimizations—sequence parallelism, causal RoPE, KV caching, and operator fusion—that enable near‑real‑time performance on modern GPUs. Ongoing work includes low‑bit quantization and further graph‑level optimizations to support even larger models and lower latency.