Advances, Challenges, and Industrial Practices in Text‑to‑Video Generation – From Diffusion Models to Sora

The rapid development of text‑to‑video ("text‑video") technology enables users to generate video content directly from textual prompts, with models like OpenAI's Sora extending generation length from a few seconds to a minute.

Diffusion models, which iteratively add and remove Gaussian noise, have become the core of modern video synthesis, offering advantages over earlier GAN or VAE approaches by modeling the generation process in multiple steps and supporting conditional inputs such as text, images, depth maps, or skeletons.

Key difficulties in video generation include: (1) realistic motion modeling to avoid unnatural or disjointed actions; (2) precise semantic alignment so that generated frames faithfully reflect detailed textual descriptions; and (3) high‑quality output that balances resolution, frame rate, and generation speed.

Tencent’s solutions address these challenges through several strategies: using an anchor frame (image condition) to stabilize motion across frames; augmenting image data to enlarge the training set; designing a multi‑resolution training framework; injecting large‑language‑model embeddings (e.g., T5, LLAMA) via cross‑attention to improve text understanding; and applying a two‑stage pipeline where the first stage generates coarse video and the second stage performs super‑resolution and targeted facial refinement.

Industrial applications demonstrated include video style transfer (e.g., real footage to anime or 3D style), human pose‑controlled animation from a single image, and a "motion brush" that lets users animate specific regions via masks and textual cues.

Future outlook highlights the disruptive impact of Sora, which leverages massive data and transformer‑based architectures to scale video length and quality, and mentions open‑source Sora projects from Chinese research groups. Tencent continues to explore transformer‑centric designs and anticipates further breakthroughs.

The Q&A section discusses evaluation metrics (CLIP similarity, expert human scoring), the importance of data volume and quality as a competitive moat, and the limits of current models in truly understanding physical world dynamics.