Honeycomb: Fixed-Size Scene Memory for Consistent Long Video Generation

Introduction

Video world models can generate visually impressive short clips, but as generation lengthens, early frames leave the model's limited temporal context. The model may "forget" previously observed spaces: furniture textures change, doors and windows shift, and room geometry drifts. A naive solution stores all past observations—RGB images, 3D points, or diffusion latent features—but memory grows continuously with video length, becoming a storage and compute burden for long-term exploration, interactive environments, or embodied AI.

Researchers from Harvard, MIT, NUS, and MIT-IBM Watson AI Lab propose Honeycomb, which answers a fundamental question: can a video world model continuously absorb new scene information while keeping feature memory at a fixed size? Honeycomb's answer is HexMemory, compressing the scene into six fixed-size 2D feature planes updated via a "read, generate, write-back" loop. Regardless of video length, HexMemory's feature storage never grows with frame count.

HexMemory Design

1. HexMemory: Three Spatial Planes, Three Spatiotemporal Planes

HexMemory uses three spatial planes (XY, XZ, YZ) and three spatiotemporal planes (ZT, YT, XT), where T denotes the write time. The six planes form three pairs, each covering the four axes X, Y, Z, T. Querying a 3D position at a specific write time involves bilinear sampling on the relevant planes and combining the features. Spatial planes store time-invariant scene structure; spatiotemporal planes capture time-varying content. This amounts to a low-rank decomposition of a 4D scene feature field, more compact than a full 4D grid or ever-growing point cache.

2. Feedforward Writer: Processing Only New Observations

The system first encodes frames into latent features via a VAE, then lifts them to 3D world coordinates using depth and camera parameters. Each 3D point carries its position, latent feature, camera center, and viewing ray direction. A lightweight network predicts feature contributions for each of the six planes, distributing information to nearby grid cells via bilinear weights. Each plane also maintains a confidence map recording how many valid observations each cell has received; these confidences later determine the weighting of old versus new information during updates.

3. Recurrent Update: Expanding Coverage Without Growing Tensors

When a new segment covers a larger spatial or temporal range, Honeycomb does not increase plane resolution. Instead, it remaps the old planes to new boundaries, letting the fixed grid cover a larger world region at the cost of lower effective resolution per unit space. The transformed old features and the current segment's new features are fused per cell by confidence weighting, then a small network predicts a residual correction. The writer processes only the new segment—no full-history reread, no per-scene gradient optimization.

Before generating the next video segment, the system projects 3D points into the target camera view, queries HexMemory, and reconstructs a latent feature map. These latents, plus a visibility mask, enter a ControlNet-like conditioning branch of the video diffusion model, allowing the generator to access scene information that has left the temporal context.

Experimental Results

Loop-Closure Revisit on WorldScore

To test whether the model truly remembers the scene, the team uses a loop-closure revisit setup: the model starts from an input image, follows a camera trajectory away from the initial position, explores other areas, then returns to the original viewpoint. The closer the final frame matches the input, the more reliable the long-term memory.

On WorldScore loop-closure tests, Honeycomb achieves the best results across all reported metrics: loop PSNR 17.22 dB (1.23 dB higher than the second-best method), SSIM 0.504, LPIPS 0.311. Optical flow error between input and final revisit frame is 3.00 pixels, versus 6.64 pixels for Spatia.

Novel View Synthesis on RealEstate10K

Long-term memory value extends beyond returning to the origin. In RealEstate10K novel view synthesis, the model starts from the first frame and continuously generates three video segments along real camera trajectories. Honeycomb achieves the best PSNR, SSIM, and LPIPS across all three metrics.

Overall Video Generation Quality

On WorldScore overall video generation quality, Honeycomb scores 65.52 average, above Spatia's 63.21 and LSM World's 61.20. Notably, it does not lead on every sub-metric; its standout advantage is maintaining strong generation quality, novel view synthesis, and revisit consistency simultaneously under a fixed feature storage budget.

Memory-Quality Trade-offs

Honeycomb's default configuration uses 512-resolution planes with feature rank 48, requiring ~73.9 MB of feature storage. Because capacity is determined by plane resolution and feature dimension—not video length—applications can choose configurations matching device resources and quality needs.

Reducing plane resolution from 512 to 256 cuts storage from 73.9 MB to 19.8 MB (~73% reduction).

Under the same setting, loop PSNR drops only from 17.22 dB to 17.10 dB, SSIM from 0.504 to 0.500.

Further reduction to 128 resolution needs only 5.8 MB, but quality degradation becomes more pronounced.

Constant Write Cost

The team compares three memory update strategies. Direct optimization yields slightly higher revisit metrics but takes ~3.2 seconds per write. Replacement-based writing reprocesses all historical observations, with write time growing from 11.2 ms to 40.1 ms as segments increase. Honeycomb's recurrent writer processes only the latest segment and fuses with old memory; write time stays stable at ~13 ms whether at the 2nd, 5th, or 9th segment—about 240× faster than direct optimization, and write cost does not grow with video length.

Discussion and Limitations

Honeycomb demonstrates a new scene memory paradigm: past information need not be stored as ever-growing images, 3D points, or latent entries; it can be continuously compressed into a fixed structure. The system processes only the latest observation yet retrieves early scene information in future views.

This design has clear boundaries. Because plane size is fixed, as the camera explores larger spatial extents and longer time spans, the fixed grid must represent increasingly larger regions, gradually reducing effective resolution per unit space. HexMemory is not "unlimited lossless" memory but a controllable trade-off among storage scale, coverage range, and detail preservation.

Honeycomb still depends on depth estimation, camera trajectories, and the base video generation model's capabilities. Errors in early geometry estimation or generation can propagate into memory updates. Distinguishing persistent structures from transient changes in dynamic scenes, and managing limited resolution over longer trajectories, remain open research questions.

Nevertheless, this work proves that long-duration video generation does not inherently require a continuously expanding scene cache. Future video world models must know where they have been, understand which scene structures should remain stable over time, and restore the same world upon revisit. Honeycomb provides a compact and efficient implementation of this ability using six fixed-size feature planes. When the model returns to the origin, it does not merely generate a plausible frame—it strives to remember: this place should still be the same place.

Paper: https://arxiv.org/abs/2609.37690 Project page: jackswl.github.io/honeycomb Code:

github.com/kaichen-z/honeycomb