LiveEdit Enables Real-Time Video Editing at 12.66 FPS via 4‑Step Denoising

Background

Text‑driven video editing requires low‑latency, continuous‑input, instant‑output processing, but high‑quality diffusion models rely on bidirectional spatio‑temporal attention that needs future frames.

LiveEdit Framework

LiveEdit processes video in a causal, chunk‑wise fashion. Inference uses four steps per chunk, achieving 12.66 FPS on an 81‑frame video while preserving edited‑region accuracy and unedited‑region consistency.

Technical Challenges

Switching from bidirectional to causal attention dilutes attention over a longer history, causing distribution shift and leading to flicker or drift in long edits.

Repeated computation on unchanged regions dominates cost because standard diffusion models apply self‑attention, cross‑attention, and feed‑forward networks to all spatial tokens each step.

Three‑Stage Distillation Pipeline

Stage 1 – Foundation Tuning : Keep the full bidirectional DiT architecture. Inject video latent variables and noise along the channel dimension (instead of extending the spatio‑temporal token sequence) to learn high‑quality editing priors.

Stage 2 – Teacher Forcing with Chunk‑wise Causal Attention : Use chunk‑wise causal attention (3 latent frames per chunk) so each chunk sees only current and past chunks. Teacher forcing forces the causal model to reproduce the editing distribution of Stage 1, gradually transferring local structural priors.

Stage 3 – Distribution Matching Distillation (DMD) : Initialize the causal generator from Stage 2 weights and train with distribution‑matching loss to compress 100 diffusion steps to 4, eliminating classifier‑free guidance.

AR‑Oriented Mask Cache

Compare the original video latent of the previous chunk with its edited result to produce a spatial mask that predicts which tokens may need editing in the current chunk. Active tokens undergo full computation; inactive tokens reuse cached self‑attention features. The threshold adapts to token redundancy, pruning ~70 % of tokens in the reported configuration.

Ablation shows self‑attention features are temporally repeatable and suitable for cross‑time reuse, while FFN features retain high‑frequency texture and cause blur if cached; therefore caching is applied only to self‑attention layers.

Performance Evaluation

LiveEdit processes 81 frames in 7.89 seconds (12.66 FPS). A 120‑sample test set evaluates six metrics: text‑instruction alignment, background consistency, motion smoothness, dynamics, aesthetic quality, and overall video quality. Qualitative results demonstrate accurate local color, material, and attribute replacement with reduced drift in non‑edited regions. The mask cache preserves background consistency without harming instruction alignment or motion continuity.

User study with 20 volunteers ranks LiveEdit in the top‑3 for instruction consistency (100 %), background retention (87.5 %), and overall quality (95.8 %).

Limitations

Assumes editing regions remain relatively stable between adjacent chunks. Rapid motion, drastic region changes, or large‑scale global edits challenge mask estimation and cache reuse. Extending to longer videos, higher resolutions, and more complex interactive commands remains open.

Paper: https://arxiv.org/abs/2606.26740

Code: https://github.com/cp-cp/LiveEdit