CineEvolve: Self-Evolving Video Generation Lets AI Directors Call 'NG' on Storyboards
Researchers from BUPT, Peking University, and BAAI propose CineEvolve, a framework where a single visual language model acts as both director and critic to plan, diagnose, and revise storyboards via alternating reinforcement learning, achieving superior multi-shot consistency and narrative coherence in generated videos.
Introduction
Generating coherent multi-shot videos requires not only creating individual shots but also ensuring global consistency across shots — entities, lighting, and narrative flow must remain stable. Real directors spot issues and call "NG" (no good) to request retakes; AI systems need a similar self-correction loop. CineEvolve , proposed by researchers from Beijing University of Posts and Telecommunications (BUPT), Peking University, and the Beijing Academy of Artificial Intelligence (BAAI), enables a single visual language model (VLM) to serve as both director (planning and revising storyboards) and critic (diagnosing defects), with both roles improving through alternating reinforcement learning.
CineTraj Dataset: Recording the Full Correction Trajectory
To teach the model how to revise storyboards, the authors built CineTraj , a dataset that captures the complete demonstration of creation, diagnosis, and revision by a teacher model (e.g., GPT-4V). The construction pipeline involves three stages:
Storyboard generation from movie scenes: Clear, narratively coherent scenes are selected from movies. The teacher model annotates the story and shot descriptions, creates a shooting plan, and calls an image generation tool to produce an initial set of storyboard frames.
Shot-level diagnosis: The teacher acts as critic, comparing generated frames against the original movie scene. It marks erroneous shots, describes specific defects (e.g., a masked character's face should be revealed but the scarf still covers it), and rates severity. As director, it decides which frames to keep, regenerate, or edit.
Full-storyboard re-check: After revisions, the critic reviews the entire storyboard again, considering both the severity of remaining issues and any new defects introduced by local edits. Only trajectories where overall defects improve are retained.
The resulting dataset contains three types of samples: initial creation, defect diagnosis, and targeted revision, used for supervised fine-tuning and subsequent RL.
CineEvolve Training: Alternating Reinforcement Learning
The director and critic share the same model parameters. Training proceeds in two phases:
Supervised Fine-Tuning (SFT)
The shared VLM is fine-tuned on the CineTraj demonstrations to acquire basic director/critic capabilities.
Alternating Reinforcement Learning
Because the director's error distribution shifts during training, the critic must adapt. Each RL round alternates:
Critic update: The critic is trained to match teacher-annotated diagnoses — identifying faulty shots, describing defects accurately, and judging severity. These diagnoses provide the basis for the director's revision plans.
Director update: The director uses the critic's diagnoses and quality scores, plus an external narrative coherence signal, to learn planning and revision. Newly generated storyboards are then annotated by the teacher model, and these fresh trajectories are added to the critic's training data for the next round, keeping diagnosis aligned with evolving director behavior.
To prevent the director from learning from revisions that actually worsen the storyboard, the authors introduce E-GRPO (Evolutionary Group Relative Policy Optimization) with two mechanisms:
Revision protection: Rewards are computed relative to the pre-revision storyboard, not just within the current candidate group.
Historical anchoring: The director's own past performance serves as an additional baseline, reducing the chance of treating regressions as improvements.
Inference
At generation time, the trained CineEvolve model autonomously switches between director and critic roles: it plans storyboards, diagnoses problems, executes revisions, and finally passes the polished storyboards to a video generation model (e.g., a diffusion-based video generator) to produce the final video.
Experiments
Comparison with Six Baselines
Using the same image and video generators, CineEvolve is compared against six existing multi-shot generation methods. Qualitative evaluation on a fishing story shows that baselines suffer from entity inconsistency (fish count changes, character clothing changes, sudden lighting shifts), while CineEvolve maintains character appearance and lighting, and advances the plot logically (casting, reeling, netting, conversation).
Quantitative results on the paper's test set show CineEvolve outperforms all baselines on three metrics:
Entity appearance consistency
Cross-shot consistency
Transition logic consistency
User preference studies across four dimensions — visual quality, text alignment, cross-shot consistency, and inter-shot coherence — also rank CineEvolve highest.
Ablation Study
Ablations on a baking story reveal the contribution of each design:
SFT only: Overall performance drops.
Fixed critic: Cross-shot consistency degrades.
Separate policies for director and critic: Narrative continuity weakens.
Standard GRPO (without revision protection and historical anchoring): Consistency and transition logic decline.
Visualizations show variants producing errors such as baked bread appearing too early, bread count changes, confusion between putting into and taking out of the oven, and sleeve changes; the full CineEvolve framework avoids these.
Limitations and Future Work
The current diagnosis and revision loop operates at the storyboard level and does not yet address fine-grained motion artifacts in the subsequent video generation stage. However, the framework establishes a concrete path for which shot has a problem and how to fix it , providing a foundation for extending self-correction to the video generation level.
Signed-in readers can open the original source through BestHub's protected redirect.
This article has been distilled and summarized from source material, then republished for learning and reference. If you believe it infringes your rights, please contactand we will review it promptly.
How this landed with the community
Was this worth your time?
0 Comments
Thoughtful readers leave field notes, pushback, and hard-won operational detail here.
