From a Page Novel to a Short Film: How Open‑AI‑Canvas Brings AI Film Production to the Browser

Overview

Open‑AI‑Canvas (named 影策) is an open‑source infinite‑canvas workbench for AI‑driven film and short‑drama creation. It brings the entire industrial workflow—storyboard → assets → generation → review → composition—into a single browser canvas, allowing creators to go from a page of text to a finished video without leaving the web.

Core Capabilities

Unlimited Canvas : Supports nine node types (Image, Text, Drawing, Script, Skill, Config, Video, Audio, Frame). Nodes can be freely linked, selected, undone/redone, and organized with a mini‑map. Import/export and read‑only sharing are provided.

Four‑Modality AI Generation : A unified queue handles text, image, video, and audio generation. Features include reference‑image editing, start‑end frame control, camera‑move parameters, video continuation, local mask editing, character texture adjustment, and transparent‑background output.

Provider Adapters : Built‑in support for 13 provider protocols—OpenAI Chat Completions, OpenAI Responses, OpenAI Images, OpenAI Audio, Volcengine image/video, Jimeng image/video, NewAPI video (three channels), xAI Video, and Google Gemini Veo. Each protocol has independent request construction, async polling, and result parsing.

Structured Storyboard Workflow : A seven‑step “short‑drama standard process” (style → story → storyboard → assets → video → review → delivery) with independent state tracking, failure retry, and step rollback.

Role Card & Asset System : Defines character appearance, three‑view sketches, voice samples, and dubbing instructions; assets bind to specific shots to maintain cross‑shot character consistency.

3D Director : Uses Three.js and @react-three/fiber to render a 3D viewport for previewing shot composition and character placement.

Architecture

The system follows a three‑layer architecture:

Browser Layer : A pure SPA built with React 19, LeaferJS for canvas rendering, Three.js for the 3D director, and TipTap for rich‑text editing.

Backend API Layer : Go + Gin provides REST APIs, classic MVC design, an async task queue, and a provider‑adapter layer. Concurrency is controlled by a two‑level limiter (worker‑level and channel‑level, default 3 each). Storage can run in single‑instance SQLite or multi‑instance PostgreSQL + Redis.

Canvas Agent Layer : An independent Node.js/TypeScript process that communicates with the browser via Server‑Sent Events (HTTP mode) or via MCP/stdio (MCP mode). It exposes 32 canvas‑operation tools and 10 project‑operation tools through @modelcontextprotocol/sdk, allowing Codex or Claude Code agents to read canvas state, create nodes, trigger generation, and manage assets.

Agent Collaboration Mechanism

Two collaboration paths are provided:

Web Canvas Assistant (built‑in) : The right‑hand assistant panel calls /api/sessions, sending user prompts and a canvas snapshot to the backend Agent. The backend uses the configured text model to produce a storyboard plan ( agentStoryboardPlan) containing title, storyline, character list, scene list, and per‑shot details (camera, mood, lighting, audio, movement, visual and video prompts).

Local Canvas Agent (MCP) : Runs as a TypeScript process. In HTTP mode it listens on 127.0.0.1:17371 and exchanges state via SSE. In MCP mode it registers 32 tools with Codex through @modelcontextprotocol/sdk and forwards tool calls to /api/tools. Security is enforced by binding to localhost, recording the first Origin, and requiring a random token for subsequent connections.

Quick Start & Deployment

Four deployment options are described:

One‑click server deployment (recommended) – runs a script that installs Docker, clones the repository to /opt/open-ai-canvas, generates random DB passwords, builds front‑ and back‑end images, and starts Web, Backend, PostgreSQL, and Redis containers.

Docker image deployment – pulls a pre‑built image and runs it.

Docker Compose – uses

docker compose -f docker-compose.local.yml up -d --build

to launch all services locally.

Local development – clone the repo, start the backend with go run ./cmd/server, and launch the frontend with Bun ( bun install && bun run dev).

After the server is up, open http://localhost:3000, register the first admin account, configure model channels, and start the Canvas Agent with npx -y @ddcat666/open-ai-canvas-agent. The agent prints its address and token; paste them into the “Agent” panel to connect. Register the agent as a Codex MCP with

codex mcp add infinite-canvas -- npx -y @ddcat666/open-ai-canvas-agent mcp

, then use tools such as canvas_get_state or canvas_generate_image directly from Codex.

References

Project repository: https://github.com/ddcat-ai/open-ai-canvas Upstream infinite‑canvas: https://github.com/basketikun/infinite-canvas Online demo: https://ddcat.pronhubcn.com Canvas Agent NPM package:

https://www.npmjs.com/package/@ddcat666/open-ai-canvas-agent