Unraveling Sora: How OpenAI Might Build a 60‑Second Video Generator

Key Messages Overview

Sora is believed to generate high‑quality 10‑60 second videos by combining a text‑to‑video diffusion model with a temporally consistent encoder‑decoder and a novel Spacetime Latent Patch mechanism.

1. Visual Encoder‑Decoder (VAE → TECO)

The author argues that Sora most likely uses a VAE‑style encoder‑decoder because almost all video generation models adopt VAE for latent compression. Two VAE families exist: continuous latent (standard VAE) and discrete latent (VQ‑VAE, VQ‑GAN). Since Sora relies on diffusion, a continuous latent space is more suitable; discrete latents would add unnecessary quantisation loss (see VQ‑VAE (Neural Discrete Representation Learning) and VQ‑GAN (Taming Transformers for High‑Resolution Image Synthesis)).

MAGVIT‑v2, a VQ‑VAE variant, is mentioned in the literature (Language Model Beats Diffusion – Tokenizer is Key to Visual Generation), but the author reasons that its 4‑frame compression would discard too much temporal detail for long‑duration video generation. Instead, the TECO model (Temporally Consistent Transformers for Video Generation) aligns better with Sora’s need for long‑range temporal information.

TECO’s architecture includes two tasks: (1) video reconstruction via a VAE encoder‑decoder and (2) MaskGit‑style token generation. For Sora, the MaskGit component can be dropped, keeping only the reconstruction pathway, and the VAE discretisation step is removed to preserve continuous latents.

2. Spacetime Latent Patch (NaViT vs. Padding)

Patchify is a second‑stage compression that splits the continuous latent map into non‑overlapping 2×2 patches, then linearly flattens them for transformer input. Smaller patches retain more detail; Sora likely uses 2×2 patches, balancing quality and compute.

To support variable resolution and aspect‑ratio videos, the author favours NaViT (Patch n’ Pack) over simple padding. NaViT treats each video frame as a variable‑length sequence of patches, requiring an attention mask that isolates each frame’s patches (see Efficient Sequence Packing without Cross‑contamination). This avoids the wasteful padding that would dominate a 2048×2048 video batch.

Each patch also receives a learned 3‑D position embedding (X, Y, time) so the transformer can reason about spatial and temporal location without relying on absolute indices.

3. Transformer Diffusion Model (Video DiTs)

The diffusion backbone replaces the conventional U‑Net with a transformer stack. The forward diffusion adds Gaussian noise to latent frames; the reverse process predicts the noise at each timestep, guided by a CLIP text encoder that maps prompts into the same latent space.

Video DiTs extend the image‑DiT design by adding two attention modules: (1) Local Spatial Attention, masked so patches only attend within the same frame, and (2) Causal Time Attention, which lets the current frame attend to a configurable history of previous frames. Stacking N such transformer blocks yields the noise‑prediction network.

Conditioning is handled by concatenating the prompt embedding and timestep embedding to each patch token (a simple yet effective approach demonstrated in VDT: General‑purpose Video Diffusion Transformers via Mask Modeling).

4. Long‑Time Consistency Strategies

Maintaining coherence over dozens or hundreds of frames is challenging. The author contrasts three strategies:

Autoregressive : each frame attends only to a short recent window (e.g., last 4 frames).

Long‑Range (FDM) : randomly sample a few distant frames as additional keys, allowing the model to reference long‑range history (Flexible Diffusion Modeling of Long Videos).

Hierarchical : first generate coarse keyframes (first, middle, last) using a global attention pass, then refine intermediate frames with both short‑range and these global cues.

Evidence from TECO’s 500‑frame benchmark (blue curve) and FDM’s red curve suggests that both approaches improve temporal consistency, and they can be combined.

5. Training Pipeline

Sora likely follows a two‑stage training regime:

Stage 1: Train a VAE encoder‑decoder on massive unlabeled image/video data (self‑supervised reconstruction).

Stage 2: Freeze the VAE and CLIP text encoder, then train the transformer diffusion model on paired <prompt, video> data. The diffusion loss predicts added noise; the model also learns position embeddings for Spacetime Patches.

To generate large amounts of high‑quality paired data, the author hypothesises Sora uses a pipeline similar to DALL·E 3: first train a Video‑Caption Model (VCM) on a curated set of <video, detailed description>, then use VCM to annotate vast video corpora, creating synthetic training pairs. This mirrors the image‑caption augmentation used for DALL·E 3.

Additionally, Sora employs a bidirectional generation scheme: during diffusion, known frames (e.g., a user‑provided image or a target ending frame) are inserted into the noise sequence with a binary mask, and the model learns to generate both forward and backward, enabling flexible tasks such as image‑to‑video, looping video, or seamless video stitching.

6. Practical Implications

The analysis concludes that Sora’s design choices—continuous‑latent VAE, TECO‑style temporal encoding, NaViT‑based patch handling, transformer‑based diffusion, and long‑range attention—collectively explain its ability to produce long, high‑fidelity videos while remaining computationally intensive. Replicating Sora would require substantial synthetic video‑caption data, a multi‑stage training pipeline, and careful engineering of attention masks to support variable‑resolution inputs.

Overall, the article provides a step‑by‑step deduction of Sora’s possible architecture, weighing alternatives, citing concrete papers (e.g., TECO (2022), NaViT (2022), FDM (2023), Video DiTs (2022)), and highlighting the trade‑offs between quality, speed, and scalability.