Edit Video on the Fly: JD’s Open‑Source Real‑Time Streaming Editor JoyAI‑Video‑Edit

What the Project Does

JoyAI‑Video‑Edit is positioned as a real‑time, open‑source video editor that processes a continuous stream of frames—either from a live camera or an uploaded clip—and applies natural‑language instructions (e.g., “turn the background into a castle”) frame by frame without waiting for the whole video.

It supports several editing modes:

Subject/character editing (costume changes, hairstyle swaps, role transformation)

Local editing (remove specific objects like two white cats on the left)

Background replacement (modern interior → Great Wall, street → seaside)

Style transfer (watercolor, oil painting, cyber‑punk)

Motion alteration (change object movement patterns)

Reference‑image guided editing (steer the output toward a supplied image)

Demo screenshots show transformations such as a modern restaurant turned into a British‑style castle banquet, water‑color rendering, dogs receiving colored hats, and selective object removal.

Technical Backbone that Enables Real‑Time

Traditional AI video editors are offline and non‑causal: they ingest the entire video, process all frames, and must re‑run the whole pipeline for any local change, causing latency.

Architecture

The system consists of three core components:

Multimodal large‑model (MLLM) encoder : parses the natural‑language command and the current frame, producing control conditions for the generator.

Causal video VAE : encodes only the current and past frames, never future frames, enabling frame‑by‑frame processing.

16‑billion‑parameter multimodal diffusion Transformer (MMDiT) : the generative backbone that produces edited frames.

These are trained as a self‑regressive diffusion editor, generating each frame sequentially to match the streaming cadence.

Speed Optimizations

To reach 30.19 FPS at 720×1280 resolution, the authors applied four acceleration techniques:

Aligned autoregressive distribution‑matching distillation (ARMD) : compresses the slow diffusion sampling into a faster distilled path.

Long‑horizon optimization : jointly optimizes consecutive generation segments to curb temporal drift in long videos.

Bounded KV‑state inference : limits the size of attention key‑value caches, preventing unbounded memory growth.

Deployment‑oriented scheduling : balances computational load across stages to sustain high‑throughput.

The reported 30.19 FPS figure is measured on the authors’ benchmark hardware; actual performance may vary with GPU, resolution, and command complexity.

Comparison with Offline Batch Editors

Offline tools (e.g., Runway, Pika) follow a “whole‑video‑in → whole‑video‑out” pipeline, offering stable results and fine‑grained iterative tweaking but requiring full rendering for any change.

JoyAI‑Video‑Edit follows a “frame‑in → frame‑out” causal streaming pipeline, delivering near‑real‑time feedback suitable for live‑stream scenarios, at the cost of higher hardware demands and ongoing engineering maturity.

Thus the two approaches serve different use cases: polished post‑production versus interactive live editing.

Getting Started and Current Limitations

The project, open‑sourced in August 2026, provides deployment code, a technical report (arXiv), an online demo, and model weights on Hugging Face. A typical setup sequence is:

# Create environment
conda create -n joyai-video-edit python=3.10 -y
conda activate joyai-video-edit
python -m pip install -r requirements.txt

# Download model weights to the designated folder
# Launch the service
cd deploy
bash run_server.sh
# Open http://localhost:8080

The demo can be tried directly without local installation.

Current barriers include the 16 B parameter model’s demand for high‑end GPUs (the authors note future support for consumer‑grade GPUs like RTX 5090) and the fact that the full training pipeline and data generation scripts are not yet released.

Who Should Use It

Researchers and developers working on video generation or multimodal diffusion.

Teams with ample GPU resources aiming to build real‑time video or live‑stream effects.

Enthusiasts who want to experiment with the online demo.

Enterprises seeking a research‑grade, open‑source foundation rather than a production‑ready commercial editor.

Key Takeaways

Real‑time performance stems from system‑level design (causal VAE, bounded KV, distillation) rather than raw compute.

Streaming generation faces cumulative drift; long‑horizon optimization mitigates this.

Frame‑level editing dramatically improves user experience compared with batch re‑rendering.

Large‑scale open‑source research systems often expose core architecture while keeping full training pipelines private.

Overall, JoyAI‑Video‑Edit pushes the frontier of “edit while you shoot” by delivering a self‑regressive diffusion pipeline that brings real‑time video editing closer to practical use.

GitHub repository:

github.com/jd-opensource/JoyAI-Video-Edit