Scal3R Enables Stable Kilometer-Scale 3D Reconstruction of Long Videos

Problem with Large‑Scale Long‑Video 3D Reconstruction

Existing 3D foundation models can estimate camera parameters, depth, and point clouds from short RGB clips, but when the sequence length grows to hundreds or thousands of frames, drift accumulates and reconstruction becomes unstable, especially in kilometer‑scale outdoor scenes with repetitive textures and sparse sampling.

Scal3R’s Core Idea

Scal3R tackles the root cause by ensuring that the model experiences long sequences during both training and inference. It integrates test‑time training (TTT) with a unified pipeline that processes long sequences, updates information chunk‑wise, and synchronizes across chunks, thereby maintaining local geometry and global consistency.

Global Context Memory (GCM)

The GCM consists of adaptive memory units that act as lightweight, updatable context modules. After each chunk is processed, a self‑supervised objective updates these units, allowing the model to accumulate and retain context across chunks and to use the same update mechanism during training and testing.

Global Context Synchronization (GCS)

GCS synchronizes the gradients of the adaptive memory units between chunks using PyTorch’s all‑reduce (DDP) mechanism, ensuring that updates are consistent across the entire sequence regardless of whether the model is training or inferring.

Why This Stabilizes Long Sequences

By processing long sequences as overlapping chunks, Scal3R avoids the quadratic cost of full‑sequence attention. Each chunk’s local geometry remains accurate, and GCM/GCS propagate reliable context, preventing error accumulation. The model therefore sees the same “long‑sequence + chunk‑update + sync” pattern during training as it does at test time.

Benchmark Results

Scal3R was evaluated on VKITTI2, KITTI Odometry, Oxford Spires, ETH3D, and other datasets. For camera pose estimation, it reduced ATE/RTE/RRE compared to VGGT‑Long (e.g., KITTI ATE 14.55 m → 4.55 m). For point‑cloud reconstruction, it achieved the lowest Chamfer Distance and highest F1 scores across all datasets (e.g., ETH3D 0.11 / 0.91 vs. 0.24 / 0.84 for VGGT‑Long). Qualitative visualizations show tighter alignment with ground‑truth trajectories and more complete structures.

Efficiency and Scalability

Running on a single RTX 4090, Scal3R’s inference time scales roughly linearly with sequence length (150 → 990 frames) while maintaining pose error around 0.07–0.08 m, demonstrating practical scalability without sacrificing accuracy.

Conclusion

Scal3R advances long‑video 3D reconstruction not by enlarging backbones or compressing tokens, but by rethinking the training‑inference alignment: training on true long sequences with updatable global context and synchronized updates, enabling stable, high‑quality reconstruction at kilometer scale.