DynStream: First Pose-Free Streaming 4D Gaussian Reconstruction for Large-Scale Dynamic Scenes

Problem: Online 4D Reconstruction from Pose-Free Video

Reconstructing a changing 3D scene while recording a long video without known camera poses presents three core challenges: incremental processing of streaming frames, maintaining global consistency across time, and achieving high-fidelity rendering of both static and dynamic elements. Existing per-scene optimization methods are too slow; offline feed-forward approaches are limited to short clips and scale poorly in compute and memory; and many feed-forward geometry models output only point clouds or depth maps, not directly renderable representations.

DynStream Framework Overview

DynStream takes a continuous RGB video stream without camera poses and produces, in a single forward pass per chunk, a camera trajectory and a continuously updated dynamic Gaussian scene representation ready for rendering. The pipeline processes video in fixed-length temporal chunks, uses a causal attention backbone for local reconstruction, aligns chunks into a unified world coordinate system without extra optimization, and accumulates static Gaussians globally while keeping dynamic Gaussians at their respective timesteps.

Module 1: Local Online Reconstruction

To avoid the quadratic attention cost of long sequences, the input video is divided into fixed-length chunks. For each incoming chunk, a causal attention backbone performs feed-forward reconstruction, predicting relative camera poses (each frame w.r.t. the chunk's first frame) and a local Gaussian representation. This design lets the network focus on accurate local geometry first, deferring global consistency to a separate aggregation step.

Module 2: Global Aggregation Without Extra Computation

Local reconstructions from different chunks live in arbitrary Sim(3) coordinate systems, causing misalignment when naively merged. Traditional methods run expensive optimization on overlapping point clouds to estimate Sim(3) transforms. DynStream eliminates this by design:

Relative pose prediction: The network outputs each frame's pose relative to the chunk's first frame, not absolute poses.

Overlapping keyframe alignment: Adjacent chunks share one overlapping keyframe. The relative poses from both chunks are used to transform the new chunk into the global coordinate system.

Per-chunk scale factor: A positive scale factor is predicted for each chunk, mapping its local scale to metric (meter) scale, removing scale drift across chunks.

These three mechanisms achieve globally consistent scene assembly with zero additional compute at inference time.

Module 3: Dynamic–Static Separation and Scene Accumulation

Each Gaussian primitive receives a predicted dynamic probability. A fixed threshold splits Gaussians into static and dynamic sets. Static Gaussians are progressively fused into a single global scene model. Dynamic Gaussians remain attached to their originating timestep. During rendering, the global static scene is combined with the dynamic Gaussians corresponding to the target timestamp, enabling coherent novel-view synthesis of moving objects without ghosting.

Experimental Results

Scene Rendering Quality (5 Datasets)

Evaluated on Waymo, SpatialVID, and three other datasets covering indoor and outdoor, static and dynamic scenes. Compared against feed-forward baselines AnySplat, MVSplat, NeoVerse, DGGT, and STORM. DynStream achieves state-of-the-art PSNR on all metrics across all five datasets. Qualitative inspection shows superior reconstruction of vehicle edges, road structure, and dynamic–static boundaries, with effective suppression of dynamic-object ghosting.

Long-Range Camera Trajectory Reconstruction (KITTI)

Tested on 11 KITTI sequences totaling 23,201 frames, with the longest sequence spanning 5 km. Compared to existing streaming methods, DynStream reduces Sim(3)-aligned ATE from hundreds of meters to 40 m (sequence-length-weighted average). Both Sim(3) and SE(3) alignment metrics reach state-of-the-art, demonstrating robust long-range trajectory estimation.

Paper: https://arxiv.org/pdf/2610.09720