Why Unified Multimodal Models Are the Key to Next‑Gen AGI – A Deep Survey

Introduction

The author argues that achieving Artificial General Intelligence (AGI) requires AI systems capable of simultaneously understanding and generating multiple modalities such as text, images, video, and audio. A panoramic overview of UFM research is presented across six dimensions: encoding, decoding, modeling, training, application, and benchmarking.

Motivation for Unification

Traditional approaches separate the "understanding" and "generation" tracks, leading to two major pain points:

Capability ceiling: tasks that need both deep comprehension and continuous generation (e.g., turning a script into a film) cannot be solved by a single pipeline.

Data and parameter redundancy: duplicated knowledge across two models causes higher latency and error accumulation.

Quoting Feynman's maxim, "What I cannot create, I do not understand," the article stresses that understanding and generation should form a mutually reinforcing loop.

What Is a Unified Multimodal Foundation Model (UFM)?

The paper formally defines a UFM as a model that can handle any task belonging to the set PowerUniSet = 2^{(T_U \cup T_G)} - 2^{T_U} - 2^{T_G}, meaning each task must contain at least one understanding component T_U and one generation component T_G. After Unified Pre‑training (UP), the model can directly produce valid outputs for any x \in I \in PowerUniSet.

Modeling Paradigms – Three Technical Routes

Route A – Plug‑in Expert : LLM acts as a scheduler that calls external black‑box APIs (e.g., Stable Diffusion, Whisper). Representative works: Visual‑ChatGPT, HuggingGPT.

Route B – Modular Joint : LLM outputs prompts or features; an external diffusion model decodes them. Representative works: NExT‑GPT, DreamLLM.

Route C – End‑to‑End Unified : All modalities are tokenized and processed by a single Transformer without external models. Representative works: Emu3, Janus‑Pro, Chameleon, BAGEL.

Encoding Strategies – Tokenizing Visual and Audio Signals

Three representation types are compared:

Continuous : CLIP/EVA‑CLIP features – good semantic alignment.

Discrete : VQ‑VAE/VQGAN codebooks – compatible with LLM vocabularies.

Hybrid : Dual‑branch (semantic + pixel) – combines strengths of both.

Decoding Strategies – From Tokens Back to Pixels or Waveforms

External Diffusion : LLM outputs are fed to frozen diffusion models (e.g., SDXL, FLUX) with lightweight adapters (e.g., Emu2, MetaMorph).

Internal Diffusion : Diffusion heads are inserted inside the LLM and trained end‑to‑end (e.g., Transfusion, Show‑o).

Discrete Autoregressive : Pure next‑token prediction without diffusion, offering faster inference at some quality loss (e.g., Emu3, Chameleon).

Three Pillars of Training UFM

Encoder‑Decoder Pre‑training : Tokenizer learns to encode and decode jointly, often by coupling VAE training or freezing CLIP and training an adapter.

Multimodal Alignment : Contrastive learning, Q‑Former, or linear projection align different modalities into a shared semantic space.

Unified Backbone Training : The backbone learns a mixed objective combining next‑token prediction (NTP), diffusion loss, and alignment loss, enabling simultaneous understanding and generation.

Fine‑Tuning and Alignment

General‑Task Fine‑Tuning : Mix multiple tasks (e.g., LLaVA‑Instruct, SEED‑Data‑Edit) with a unified NTP loss.

Multi‑Task Fine‑Tuning : Domain‑specific data such as medical imaging or 3D point clouds, using staged or expert‑wise training to mitigate task conflicts.

Human‑Preference Alignment : DPO/GRPO triples provide joint rewards for understanding + generation, applied via iterative SFT → DPO.

Data Engineering – "Garbage In, Garbage Out"

The survey breaks data pipelines into four sources, four cleaning steps, and three construction methods:

Sources: public crawls (LAION‑5B), curated annotations (COCO), private datasets, synthetic data (GPT‑4o).

Cleaning: deduplication → NSFW filtering → aesthetic scoring → CLIPScore filtering.

Construction: (a) rewrite existing datasets as <instruction, input, output>; (b) use large models to synthesize complex instructions; (c) human‑verified fine‑grained labels + crowd‑sourced preferences.

Benchmarking – Fair "Horse Racing"

Benchmarks are categorized into three dimensions:

Understanding : MMBench, MMMU, MathVista – fine‑grained skills with multi‑choice and automatic scoring.

Generation : GenEval, T2I‑CompBench, VE‑Bench – evaluate composition, editing, and physical consistency.

Mixed : MME‑Unify, RealUnify – first benchmarks requiring mutual reinforcement between understanding and generation.

Real‑World Applications

Unified models unlock new capabilities across domains:

Robotics : Video‑to‑world‑model pipelines (e.g., GR‑2, SEER) enable zero‑shot generalization.

Autonomous Driving : Joint future‑frame and trajectory prediction (DrivingGPT, Epona) reduces redundant perception heads.

World Models : 4D diffusion (Aether, TesserAct) learns physics from video + depth + pose.

Medical : LLM‑CXR, HealthGPT generate reports from X‑rays and can reconstruct images from textual reports.

General Vision : VisionLLM v2, VGGT provide unified detection, segmentation, depth, and 3D reconstruction without task‑specific heads.

Future Directions

Modeling : AR + Diffusion hybrids remain dominant; Mixture‑of‑Experts routing needs finer granularity.

Tokenizer : Move toward an "Omni‑Tokenizer" that handles all modalities with a single codebook.

Training : Fine‑grained interleaved data + reinforcement learning from human preferences, using dual‑task reward functions.

Evaluation : Quantify the bidirectional boost between understanding and generation rather than relying on isolated metrics.

https://www.techrxiv.org/doi/pdf/10.36227/techrxiv.176289261.16802577
A Survey of Unified Multimodal Understanding and Generation: Advances and Challenges
https://github.com/BradyFU/Awesome-Multimodal-Large-Language-Models/tree/Unified