How ViMax Turns a Simple Idea into a Full AI‑Generated Video in Minutes

Problem with Existing AI Video Tools

Typical AI video generators produce only short clips, suffer from inconsistent character appearance across frames, and require extensive prompt engineering and manual post‑processing. These limitations make end‑to‑end video creation time‑consuming and technically demanding.

ViMax Overview

ViMax is an open‑source (MIT‑licensed) multi‑agent video generation framework released by a Hong Kong University research team. It automates the full video production pipeline—script writing, storyboard creation, style control, video rendering, and audio synthesis—by orchestrating four specialized AI agents. Users provide a high‑level idea (or a novel, or a script) and receive a coherent video with sound.

Architecture

Scriptwriter Agent : Generates a structured screenplay, including scene descriptions, dialogue, and narration.

Director Agent : Converts the screenplay into a storyboard, selects camera angles, and defines shot transitions.

Producer Agent : Applies a global visual style (e.g., cartoon, Ghibli, cyber‑punk) and locks character/scene attributes using an internal reference‑image system to ensure consistency.

Post‑Production Agent : Renders the video frames, synthesizes voice‑over and background sound effects, and assembles the final MP4 file.

Key Technical Features

Idea‑to‑Video Modes : Supports "idea→video", "novel→video" and "script→video" pipelines. The system expands vague prompts into full scripts, storyboards, and rendered video without user‑written code.

Reference‑Image Consistency : After generating the first frame, a reference image is stored; subsequent frames are conditioned on this image, reducing character/scene drift by roughly 80 % compared to single‑agent generators.

Modular Multi‑Agent Collaboration : Each agent operates on a dedicated task and communicates via shared data structures, enabling richer output (dialogue, sound effects, transitions) while keeping the overall workflow simple.

Customizable Style Templates : Users can add new style definitions (e.g., ink‑wash, vintage film) and adjust camera parameters such as focal length, shot duration, or slow‑motion.

Installation

Clone the repository and install dependencies (recommended uv to avoid version conflicts):

# Clone the project
git clone https://github.com/HKUDS/ViMax.git
cd ViMax
# Install dependencies
uv sync

Configure API keys for the large language model, image generator, and video generator in configs/idea2video.yaml. Example snippet:

chat_model:
  init_args:
    model: google/gemini-2.5-flash-lite-preview-09-2025
    api_key: YOUR_API_KEY
    base_url: https://openrouter.ai/api/v1
image_generator:
  api_key: YOUR_API_KEY
video_generator:
  api_key: YOUR_API_KEY

Edit main_idea2video.py to specify the high‑level idea, audience constraints, number of scenes, and optional style:

idea = "A kid and a robot conduct a science experiment and launch a tiny rocket"
user_requirement = "Suitable for kids, 2 scenes, cartoon style, simple dialogue"
style = "Cartoon"  # optional: Ghibli, cyber‑punk, etc.

Run the generation script:

python main_idea2video.py

Generated Output

All artifacts are stored under .working_dir/idea2video: script.txt – the full screenplay produced by the Scriptwriter agent. storyboard/ – a folder of storyboard images with annotated camera angles. final_video.mp4 – the rendered video with synchronized background music and voice‑over.

Representative Use Cases

Self‑Media Content Creation : A creator supplied a one‑sentence idea and obtained a 1 min 30 s cartoon video (script, storyboard, and audio) in under an hour.

Novel Adaptation : The "Novel2Video" mode split a short sci‑fi story into three episodes, preserving core plot points and character designs while automatically adding narration and ambient sound.

Personal Cameo (AutoCameo) : By uploading a portrait, the system generated a video where the user’s likeness was integrated into a fairy‑tale scene with natural motion and dialogue.

Project Repository

https://github.com/HKUDS/ViMax