FFGo: Turning the First Frame into a Conceptual Memory for Video Customization

Background

Recent text‑to‑video and image‑to‑video models treat the first frame as a simple start. Studies from the University of Maryland, USC and MIT reveal that the first frame functions as a "conceptual memory buffer" that silently stores objects, textures, and layouts for all subsequent frames.

Key Insight

Video generation models automatically remember every visual entity present in the first frame and reuse them later, effectively encoding a conceptual blueprint of the entire video.

FFGo Method

FFGo activates this latent multi‑object fusion ability with a lightweight pipeline:

Automatic training‑set construction: a vision‑language model (Gemini‑2.5 Pro) detects foreground objects, SAM‑2 extracts RGBA masks, and captions are generated to create 20–50 high‑quality video examples.

Few‑shot LoRA fine‑tuning: a LoRA adapter is trained on the curated examples using a special <transition> prompt that acts as a transition signal to trigger multi‑object fusion.

Inference adjustment: the first four compressed frames of the base model (e.g., Wan2.2) are discarded, so generation starts from the fifth frame where the fused content appears.

The procedure does not modify the underlying model architecture and typically completes within a few hours on a single GPU.

Advantages

No changes to model structure.

Only 20–50 carefully curated examples are required, instead of millions of samples.

Training time is a few hours.

Achieves state‑of‑the‑art video customization quality.

Experimental Results

FFGo was evaluated on several video models (Veo‑3, Sora‑2, Wan2.2) and compared against VACE and SkyReels‑A2. Key findings:

Preserves object identity across frames.

Handles up to five reference entities (versus three for competing methods).

Avoids catastrophic forgetting typical of full‑model fine‑tuning.

Produces more natural and temporally coherent videos, especially in multi‑object and interaction scenarios.

Understanding Model Behavior

Baseline models occasionally generate perfect multi‑object videos, indicating the capability already exists but is unstable and hard to reproduce. FFGo’s LoRA does not teach new abilities; it learns how to reliably trigger the existing latent memory mechanism.

Conclusion

FFGo shows that video models inherently possess a powerful multi‑object fusion ability stored in the first frame. By using a small, automatically curated dataset, a transition prompt, and few‑shot LoRA, this hidden skill can be activated consistently without degrading the original generative quality.

Paper: https://arxiv.org/abs/2511.15700

Project page: http://firstframego.github.io