Stream3D Enables Streaming 3D Reconstruction and Generation Without Retraining

Problem Context

Recovering a complete and stable 3D object from an open‑ended monocular video requires both faithful observation stitching (3D reconstruction) and prior‑driven completion (3D generation). Existing view‑conditioned 3D generators assume a single image or a fixed set of images, which fails for continuous video streams.

Three naïve strategies are identified:

Generate each frame independently – yields temporally inconsistent shapes because the model lacks memory of past observations.

Retain and repeatedly process all historical frames – memory and compute grow linearly with video length, making long streams infeasible.

Pass latent states (e.g., KV‑Cache or FlowEdit) – latent alignment across views is unreliable and errors accumulate over long sequences.

Stream3D Overview

Stream3D adds a training‑free streaming mechanism to a frozen view‑conditioned 3D generator. It continuously selects reliable multi‑view evidence from the video, stores it in a fixed‑capacity evidential memory, and uses the selected frames as conditions for the generator.

Key Components

1. Attention Probe – During a lightweight pre‑heat forward pass, Stream3D reads the frozen generator’s cross‑attention maps. For each 3D query token it computes an Evidence Score that reflects how strong and distinctive the current view’s observation is for that spatial region.

2. Adaptive Evidence Memory – Each token maintains two fixed‑size lists: the top‑D evidence scores and the corresponding frame indices. When a new video chunk arrives, a frame replaces an entry only if its evidence score exceeds the stored one. This yields two properties:

Memory size is O(Q·D) and independent of total video length T.

Evidence scores for each token are monotonic (non‑decreasing) within the memory.

3. Evidence‑Based Multi‑View Generation – In the full generation stage, each token votes for the historical frames that best explain it. The system aggregates votes, selects the top‑K frames with highest token coverage, and feeds those frames as conditions to the frozen generator. Token‑wise voting allows different object surfaces to be reconstructed from different historical viewpoints.

Experimental Evaluation

Experiments on the GSO and NAVI datasets compare Stream3D against strong baselines (SAM‑3D, TRELLIS, Hunyuan3D, and the streaming reconstruction model StreamVGGT). Metrics include Chamfer Distance (CD), Intersection‑over‑Union (IoU), PSNR, LPIPS, and Image FID.

Results on GSO:

CD reduced from 0.094 to 0.048 (‑48.9%).

IoU increased from 0.664 to 0.775 (+11.1 pp).

PSNR improved from 14.178 dB to 16.145 dB.

LPIPS decreased from 0.178 to 0.139 (‑21.9%).

Image FID dropped from 105.197 to 66.711 (‑36.6%).

On NAVI (more complex camera trajectories and self‑occlusion), Stream3D still achieved the best scores across all reported metrics, e.g., CD from 0.138 (SAM‑3D) to 0.128, PSNR from 16.159 dB to 16.474 dB, Image FID from 141.496 to 134.025.

Qualitative comparisons show more consistent shapes and textures for shoes, toys, and animal models.

Ablation Studies

Deeper Transformer features provide more stable geometric evidence.

Normalized evidence scores outperform raw entropy‑based scores.

Increasing memory depth improves long‑range consistency, but overly deep memory can re‑introduce weak or outdated evidence.

K = 8 offers a good trade‑off between efficiency and quality; K = 16 yields modest further gains.

Comparison of Streaming Strategies

KV‑Cache carries all historical tokens indiscriminately, leading to unfiltered noise. FlowEdit excels at short‑range consistency but cannot preserve indispensable long‑range observations. Random frame selection may miss critical viewpoints. Stream3D’s per‑token evidential memory retains locally indispensable observations over long sequences, resulting in superior quantitative and qualitative performance.

Discussion and Limitations

Stream3D demonstrates that a generative model constrained by continuous, evidence‑driven inputs can match or exceed pure reconstruction systems in both geometry and appearance. The approach does not replace reconstruction; it bridges observation accumulation with prior‑driven completion.

Limitations: performance depends on the underlying frozen generator’s ability to interpret individual views. If the base model fails to extract correct geometry from any view, the evidential memory cannot compensate.

Future directions include fully integrating reconstruction and generation so that models continuously understand observed scenes while hallucinating unseen parts.

Paper: https://arxiv.org/abs/2605.21472

Code: https://github.com/kaichen-z/STREAM3D