Unlocking Multimodal Intelligence: A Deep Dive into Next Token Prediction

Introduction

The past two years have seen a surge of multimodal models built on the Next Token Prediction (NTP) paradigm, referred to as MMNTP. These models have achieved notable progress on both multimodal understanding (e.g., LLaVA, QwenVL) and generation tasks (e.g., Unified‑IO series, Chameleon, VAR, Transfusion, MAR, Moshi for audio).

Multimodal Tokenization

Tokenization is the cornerstone of MMNTP, converting visual, video, and audio signals into sequences of tokens that Transformers can process. Two main approaches exist: discrete tokenization, which quantizes data into a finite codebook, and continuous tokenization, which preserves raw continuity. The figure below illustrates both methods.

Tokenizer Training Methods

For image, video, and audio modalities, common training strategies such as contrastive learning and auto‑encoding are surveyed. The discussion focuses on representation and reconstruction capabilities, highlighting challenges like codebook collapse in discrete encoders and semantic alignment in continuous encoders (e.g., CLIP). Improvements such as FSQ, LFQ, and modality‑specific adaptations are also covered.

MMNTP Model Architectures

Typical MMNTP systems consist of a backbone Transformer, modality‑specific tokenizers, and de‑tokenizers. Models are categorized into compositional (leveraging external encoders/decoders like CLIP and SD3) and unified (using lightweight encoders/decoders such as VQVAE). The unified design enables a single architecture to handle diverse tasks—visual question answering, image generation, and instruction‑guided editing—by simply reconfiguring input‑output token streams.

Training Paradigms

Task Types

Training tasks are split by the nature of predicted tokens: discrete token prediction (often paired with language model heads for understanding tasks) and continuous token prediction (paired with diffusion heads for generation). This distinction dictates model heads and loss functions.

Training Stages

MMNTP training mirrors language model pipelines: (1) multimodal‑language alignment pre‑training on image‑text pairs, (2) instruction fine‑tuning for downstream tasks, and (3) preference learning to align outputs with human judgments—an emerging research direction.

Prompt Engineering

Effective prompting boosts performance. Two strategies are surveyed: multimodal in‑context learning (injecting task examples) and multimodal chain‑of‑thought (adding reasoning cues such as "perception" and "inference"). Tables summarize representative methods.

Datasets and Evaluation

The survey details dataset construction, scale, and diversity for MMNTP training. Comparative results show NTP models outperform non‑NTP baselines on large‑scale understanding benchmarks (VQAv2, MMMU) and achieve parity or superiority on generation benchmarks (ImageNet, GenEval), highlighting their unified capability.

Open Challenges

Leveraging unsupervised multimodal data to scale MMNTP.

Mitigating multimodal interference while enhancing cross‑modal synergy.

Improving training and inference efficiency.

Extending MMNTP as a universal interface for broader tasks.

Addressing these challenges is crucial for advancing multimodal intelligence.

Conclusion

By adopting a bottom‑up NTP perspective, this survey provides a comprehensive map of recent advances—from tokenization to architecture, training, and evaluation—aiming to guide researchers toward the next breakthroughs in multimodal AI.

Paper: https://arxiv.org/abs/2412.18619

GitHub repository: https://github.com/LMM101/Awesome-Multimodal-Next-Token-Prediction