World-T³: Persistent Memory for Real-Time Video World Models via Test-Time Training

Introduction

Video world models generate navigable virtual environments from a reference image and camera poses. However, autoregressive diffusion models lose memory once past frames exit the attention context window, causing scene inconsistency during long interactions.

World-T³ Approach

Researchers from Meta Reality Labs, University of Illinois Chicago, and Brown University propose World-T³ (Test-Time Training), a camera-controllable video world model that uses test-time training to store "camera pose → scene content" mappings in fast weights. When the camera revisits a pose, the model retrieves the stored scene features.

Key Results

On a 120-trajectory "out-and-back" benchmark (10 seconds each, 160 frames at 16 FPS, 1280×704), World-T³ improves revisit PSNR from 13.52 dB (SANA-WM) to 18.31 dB and reduces absolute rotation error from 3.344° to 1.075°.

A 4-step distilled model runs at 20.6 FPS on a single NVIDIA B200 GPU at 1280×704 resolution, with memory updated in real time.

World-T³ ranks first on all six revisit consistency metrics compared to Matrix-Game 3.0, HY-World 1.5, DreamX-AR, LingBot-Fast, and SANA-WM.

Method Details

Stage 1: Bidirectional Camera-Conditioned Teacher

Starting from the pretrained Wan2.2-TI2V-5B model, camera poses are encoded as Plücker raymaps (6-channel ray direction and moment fields aligned with latent grid). These are injected into each DiT block via AdaLN (scale + shift). Bidirectional attention lets the model see the whole sequence, learning to follow camera trajectories coherently. This teacher initializes the next stage and provides a reference score for distillation.

Stage 2: Autoregressive Teacher with TTT Memory

The bidirectional model is converted to a block-causal autoregressive model with persistent memory. Context for generating block b consists of: (1) the first generated block as a permanent "sink" for global appearance, and (2) the most recent K clean blocks as a sliding window for local coherence. Context length is fixed at 21 latent frames.

History outside the window is handled by a TTT memory module in every DiT layer. Each memory is a two-layer MLP: a fixed key projection W₁ maps pose raymaps to a nonlinear feature space; a readout matrix W₂ accumulates scene representations. Keys come from camera raymaps; values come from clean (denoised) hidden states, decoupling memory from diffusion noise.

When a block exits the context window, its raymap and hidden state are encoded into a key-value pair and "consolidated" into memory via a weighted ridge regression solved online with Recursive Least Squares (RLS). The RLS update uses the inverse cumulative pose-feature correlation matrix P as a whitening preconditioner, writing only the novel residual orthogonal to already stored directions. Because W₁ is fixed, each update yields the exact minimum of the objective, making the final memory state order-invariant.

At retrieval, the target block's raymap queries the memory via a forward pass; the result is injected into the layer's hidden state through a per-channel gated residual connection. During training (outer loop), gradients flow through the recursive updates, jointly optimizing key projection, base readout matrix, encoders, sample weights, and gating parameters. At deployment, the backbone and encoders are frozen; only W₂ and P evolve online, with no backpropagation through the diffusion transformer.

Stage 3: Dual-Teacher Distribution Matching Distillation

The 50-step autoregressive teacher is distilled to a 4-step generator. Consistency distillation first discretizes the continuous flow matching into 4 steps. Distribution Matching Distillation (DMD) then recovers quality and consistency lost by the large step jumps.

Unlike single-teacher DMD, World-T³ uses two complementary teachers: (1) the bidirectional teacher evaluates short-window visual fidelity and motion coherence; (2) the TTT-augmented autoregressive teacher evaluates long-horizon spatial consistency along the full sequence. Each teacher has a structurally identical critic network trained on student-generated samples. Crucially, before computing the memory score, both teacher and critic memories are updated on the student's own generation rollout, directly penalizing trajectory drift and inconsistency accumulated in the student's rollout.

Engineering Optimizations: 20.6 FPS Real-Time Generation

Without changing model architecture or numerical precision, inference optimizations yield a 3.12× speedup: reuse memory reads and RoPE-rotated context keys across the 4 denoising steps; reuse the invariant context prefix; apply torch.compile, FlashAttention-4, cuDNN 9, channels-last VAE decoding, automatic convolution engine selection, and background noise prefetching. End-to-end time for a 30-second video drops from 70.48 s to 22.59 s; per-block latency falls from 1855 ms to 589 ms. Numerical equivalence between optimized and original implementations is verified at both operator and full-generation levels.

Experimental Results

Evaluation Setup

120 unseen "out-and-back" test segments (10 s, 160 frames, 16 FPS, 1280×704) are constructed; the camera reverses at frame 80. Revisit metrics compare frame pairs with identical poses but temporal gaps of 24–36 latent frames, all exceeding the 21-frame context window. Metrics include DINOv2 semantic similarity, PSNR vs. ground truth, camera tracking error (via VGGT-Ω reconstructed trajectories), and VBench visual quality.

Revisit Consistency Leads by a Wide Margin

Compared to the strongest baseline SANA-WM, 4-step World-T³ nearly halves DINOv2 feature distance (0.0800 → 0.0390) and raises PSNR by 4.79 dB (13.52 → 18.31 dB). Per-segment paired tests show World-T³ achieves lower semantic distance on 111/120 segments and higher PSNR on 112/120 segments; on 96 segments it beats all baselines on both metrics simultaneously.

More Accurate Camera Following

Absolute rotation error drops from 3.344° (SANA-WM) to 1.075°; camera matrix error from 0.1200 to 0.0318. Some baselines show low neighbor-frame error but high absolute error, indicating cumulative drift; World-T³ maintains low error at both local and global scales.

Visual Quality on Par

VBench average score across 7 dimensions is 74.52, within the same range as baselines (72.87–75.32) and real video (75.69).

Distillation Preserves Memory

After 50→4 step distillation, revisit PSNR slightly improves (16.69 → 18.31 dB), exceeding the teacher on 91/120 segments. Four camera metrics show no statistically significant difference vs. teacher. Trade-off: VBench drops marginally from 75.15 to 74.52.

Limitations and Conclusion

The authors note two limitations: (1) reliance on 3D foundation models like VGGT-Ω for camera trajectories — pose estimation errors affect pose-based memory retrieval; (2) visual detail may degrade when generation length far exceeds training length.

World-T³ demonstrates that persistent memory need not grow unbounded; test-time training can write the observed world into model parameters, with RLS ensuring each write adds only novel information. Combined with dual-teacher few-step distillation, this achieves both long-term memory and real-time generation on a single GPU, offering a memory-efficient path for interactive world models.