I2V-Adapter: A Lightweight Image‑to‑Video Adapter for Stable Diffusion Video Diffusion Models

Research Background The task of image‑to‑video (I2V) generation faces challenges in extracting temporal dynamics from a single static image while preserving realism and visual continuity. Existing methods often require extensive model modifications and large training datasets, leading to high computational costs.

Research Plan The authors propose I2V‑Adapter, a lightweight adaptation module for Stable Diffusion‑based video diffusion models. It injects the input image as the first video frame into the spatial self‑attention layers, using zero‑initialized output mappings and training only the output and query projection matrices. A Frame Similarity Prior and a Content‑Adapter (IP‑Adapter) are added to enhance temporal consistency and semantic understanding.

Business Application The module has been open‑sourced (GitHub) and accepted at SIGGRAPH 2024. It enables fast, high‑quality video generation for various scenarios, including personalized T2I, ControlNet‑guided generation, and integration with MediaTek’s Dimensity platform for on‑device inference.

Outlook I2V‑Adapter demonstrates plug‑and‑play compatibility, allowing seamless integration with DreamBooth, LoRA, and ControlNet, and promises further advances in efficient, controllable video generation across diverse applications.