Open-Sora 2.0: How an 11B Open-Source Model Beats Closed-Source Video AI at 720p

Open‑Sora 2.0 Overview

Open‑Sora 2.0 is an open‑source video generation model with 11 B parameters that generates 720p (1280×720) video at 24 fps. It achieves visual fidelity, text‑image consistency, and motion smoothness comparable to leading proprietary models such as HunyuanVideo and Step‑Video, as measured by VBench.

Key Specifications

Parameters: 11 B

Resolution: 720p (1280×720)

Frame rate: 24 fps

Training cost: ≈ $200 k (≈ 1/5 of comparable closed‑source projects)

Hardware: 224 GPU cards (e.g., A800/RTX 3090 class)

Technical Contributions

Spatio‑Temporal Compression Suite : A 3‑D auto‑encoder combined with Flow Matching reduces spatial resolution by 8× and temporal length by 4× while preserving motion continuity.

MMDiT architecture : Integrates textual prompts directly into the diffusion process, eliminating mismatches between text and generated frames.

Multi‑Bucket Training + Parallel Optimization : Starts with low‑resolution warm‑up, progressively scales to full resolution, and leverages ColossalAI’s tensor‑parallel and pipeline‑parallel strategies, cutting training cost by 5–10×.

Training Pipeline

The pipeline is fully distributed and reproducible. It uses three stages: (1) pre‑training a 3‑D auto‑encoder on video frames, (2) initializing the diffusion model with weights from the open‑source FLUX video model, and (3) fine‑tuning on the target dataset with the multi‑bucket schedule. The entire process is scripted in the repository.

Open‑Source Release

The GitHub repository ( https://github.com/hpcaitech/Open-Sora) provides:

Model weights for the 11 B checkpoint.

Inference code supporting batch generation and adjustable parameters such as motion amplitude and aesthetic score.

A one‑click Gradio demo for interactive testing.

Full training scripts, configuration files, and documentation for reproducing the distributed training on a GPU cluster.

Performance Evaluation

VBench evaluation shows Open‑Sora 2.0 surpasses most open‑source SOTA video generators on visual quality, text‑image alignment, and motion smoothness, and reaches parity with high‑budget proprietary models (e.g., Step‑Video, HunyuanVideo).

Limitations and Future Work

While the 11 B model handles many scenes well, it may struggle with highly complex or long‑duration content. Ongoing work focuses on a higher‑compression video auto‑encoder that reduces single‑card inference time to ~3 minutes for 5‑second clips (≈ 10× speed‑up) and on extending style diversity (e.g., 3D realistic, cyber‑punk) in future releases such as Ocean V2.0.

References

GitHub:

https://github.com/hpcaitech/Open-Sora