Real-Time Frame Model (RTFM): Single‑GPU World Model Redefines 3D Generation

World Labs, the startup founded by Stanford professor Fei‑Fei Li, announced a new breakthrough: the Real‑Time Frame Model (RTFM), a generative world model that can run on a single H100 GPU.

Last month the company released Marble, a spatial‑intelligence model that creates persistent 3D worlds from a single image. Today RTFM extends this capability by producing a continuously consistent 3D experience in real time.

RTFM (Real‑Time Frame Model) does not construct an explicit 3D representation. Instead, it takes one or more 2D images as input and directly generates new 2D views of the same scene from different viewpoints.

Technically, RTFM is a learning‑based renderer: an end‑to‑end trained autoregressive diffusion Transformer that operates on frame sequences. It is trained on massive video datasets, allowing it to implicitly learn 3D geometry, reflections, shadows, and other visual effects simply by observing samples.

It can also reconstruct real‑world scenes from sparsely captured photos.

World models require massive compute

World models aim to reconstruct, generate, and simulate persistent, interactive, physically accurate environments in real time. Recent advances in generative video modeling are extending into generative world modeling, but the computational demands will far exceed those of current large language models.

For example, generating a 4K, 60 fps interactive video stream would require outputting over 100 000 tokens per second—roughly the length of an entire novel. Maintaining consistency over an hour‑long interaction would demand processing more than 100 million tokens of context, which is infeasible with today’s hardware.

According to Rich Sutton’s “The Bitter Lesson”, methods that scale gracefully with compute dominate AI research. Generative world models fit this trend: as compute costs continue to drop, they will become increasingly practical.

This raises the question: are generative world models limited by current hardware, or can we already preview their capabilities?

World Labs set out to design an efficient, deployable model that runs on a single H100 GPU while remaining scalable as compute grows. Their goal is an interactive frame rate and a world that persists regardless of interaction length.

Scalability: World models as learnable renderers

Traditional 3D rendering relies on explicit representations such as meshes or point clouds, which are hard to scale. RTFM takes a different approach: it uses a neural network trained on large‑scale video data to predict the next frame given previous frames, without any explicit 3D structure.

The input images are transformed into neural activations (KV cache) that implicitly encode the entire world. When generating a new frame, the model attends to this representation to produce a view consistent with the input perspective. This mechanism is learned end‑to‑end from data, allowing the model to capture complex effects like reflections and shadows.

RTFM blurs the line between reconstruction and generation. With many input viewpoints, the task resembles reconstruction; with few viewpoints, the model must infer missing views, behaving more like generation.

Persistence is another key property: the world should remain stable when the user looks away and return to previously visited locations. For autoregressive frame models, maintaining persistence is challenging because each new frame adds computational cost, limiting the size of the world that can be remembered.

RTFM addresses this by modeling each frame’s pose (position and orientation) in 3D space, and by using a context‑juggling mechanism that keeps geometric structure while remaining efficient, enabling true world persistence in large scenes.

If you haven’t tried RTFM yet, you can experience it at https://rtfm.worldlabs.ai/ . A podcast about the model is available at https://www.worldlabs.ai/blog/rtfm .