How Omni-Effects Enables Spatially Controllable Multi‑VFX Generation with LoRA‑MoE

Overview

Visual effects (VFX) are essential in modern film production, but existing video‑generation models are limited to single‑effect generation because each effect requires a dedicated LoRA adapter, preventing spatially controllable composite effects. This limitation hampers applications that need multiple effects simultaneously at specific locations.

To address these challenges, the authors propose Omni-Effects , the first unified framework that can generate prompt‑guided, spatially controllable composite VFX. The framework’s core innovations are:

LoRA‑MoE (LoRA expert mixture model) : integrates a set of expert LoRAs within a single model to handle multiple effects while reducing cross‑task interference.

Spatial‑Aware Prompt (SAP) with an Independent Information Flow (IIF) module: embeds spatial mask information into text tokens for precise pixel‑level control and isolates control signals for each effect to avoid unwanted mixing.

To support this research, a novel data‑collection pipeline combining image editing with first‑last‑frame‑to‑video (FLF2V) synthesis was used to build the comprehensive VFX dataset Omni‑VFX , accompanied by a dedicated VFX evaluation framework.

Approach

The strength of Omni‑Effects stems from the two core technologies:

LoRA‑MoE : By coupling LoRA with a mixture‑of‑experts architecture, each “effect expert” specializes in a particular visual effect, mitigating the interference that occurs when multiple LoRAs are jointly activated.

SAP‑IIF : The Spatial‑Aware Prompt injects spatial mask data into the Transformer’s attention mechanism, achieving pixel‑accurate effect placement without the heavy parameter overhead of methods like ControlNet. The IIF acts as a firewall, ensuring that control signals for different effects remain independent.

Experiments

Quantitative comparisons show that LoRA‑MoE outperforms baseline LoRA setups, and the controllable VFX results demonstrate precise spatial control and higher visual fidelity. Visual examples illustrate the diversity and quality of generated effects.

Conclusion

Omni‑Effects provides a unified framework for generating customized VFX videos, supporting single, multiple, and spatially controllable multi‑VFX generation. By integrating LoRA‑MoE and SAP‑IIF, the system mitigates cross‑condition interference and ensures precise spatial control. The accompanying Omni‑VFX dataset and evaluation suite validate the method’s robustness, opening new possibilities for film, game, and advertising industries.

Paper: Omni-Effects: Unified and Spatially Controllable Visual Effects Generation

Project: https://github.com/AMAP-ML/Omni-Effects