CoDeF: A Canonical Content Field Approach for Consistent Video Processing

Recently, a video‑processing algorithm called CoDeF created by Ant Group’s Interactive Intelligence Lab surged to the top of GitHub’s Python trend list within a week. CoDeF can effortlessly perform video style transfer, video key‑point tracking (including fluids), and user‑defined video content editing.

Experiments show that CoDeF upgrades image‑style algorithms to video‑style, image key‑point detection to video‑key‑point tracking (even for non‑rigid objects like water and smoke), image semantic segmentation to video object tracking, and image super‑resolution to video super‑resolution, all while supporting interactive video editing.

Current mainstream video‑generation methods suffer from poor temporal consistency, limiting their applicability in real‑world scenarios.

To address this, the researchers propose simplifying video processing to image processing. They represent a video as a 2‑D canonical content field (containing all texture information) and a 3‑D temporal deformation field (modeling motion). Each frame is obtained by deforming the canonical image with the deformation field, allowing static image algorithms to be applied frame‑by‑frame and automatically propagated through time, thus guaranteeing high temporal consistency.

The team also minimized the domain gap between canonical and real images, enabling existing image algorithms to be used on canonical images without additional training.

After being open‑sourced on GitHub, CoDeF attracted widespread attention on Twitter, with users calling it a "huge leap" and predicting its use in film production within a year.

The project was completed in three months by researchers from Ant Group’s Interactive Intelligence Lab, led by Shen Yujun, with contributions from HKUST PhD student Ouyang Hao, Ant researcher Wang Qiuyu, and Zhejiang University PhD student Xiao Yuxi.

Since its establishment in 2021, Ant’s Technology Research Institute has focused on foundational AI research, including computer vision and multimodal understanding, developing general AI algorithm architectures for content generation, digitalization, and human‑machine interaction.