What Drives Alignment in Multimodal Large Language Models? A Comprehensive Review

Introduction

Multimodal large language models (MLLMs) extend the capabilities of text‑only LLMs to process images, video, and audio. Despite these extensions, MLLMs still exhibit hallucinations, safety vulnerabilities, and limited multimodal reasoning. Alignment algorithms have been proposed to steer model behavior toward human preferences and to address these shortcomings.

Application Scenarios and Representative Methods

General image understanding : Early alignment works such as Fact‑RLHF, DDPO, HA‑DPO, and mDPO focus on reducing hallucinations and improving safety, dialogue, and reasoning abilities.

Multi‑image, video, and audio tasks : Methods like MIA‑DPO, LLaVA‑NeXT‑Interleave, and Video‑SALMONN 2 construct preference data for complex multimodal inputs, enabling better performance on multi‑image reasoning, video understanding, and audio‑visual integration.

Domain‑specific extensions : Specialized approaches such as 3D‑CT‑GPT++ (medical imaging), MAVIS (mathematical reasoning), AdPO and VLGuard (robustness), and INTERACTIVECOT/EMMOE (agent‑oriented tasks) adapt alignment techniques to particular domains.

Alignment Data Construction

Alignment datasets are essential and can be grouped into two broad categories:

External‑knowledge datasets : Human‑annotated data and data generated by closed‑source models (e.g., GPT‑4) provide high‑quality signals at high cost. Representative examples include LLaVA‑RLHF (≈10 k human‑labeled samples) and LRV‑Instruction (≈400 k visual instructions generated by GPT‑4).

Self‑annotated datasets : Models generate their own preference pairs across single‑text, single‑image, and image‑text modalities. Techniques such as SQuBa (negative‑sample generation), Image DPO (visual perturbations), and AdPO (adversarial image‑text pairs) illustrate this approach.

Balancing data quality, scale, and cost remains a central trade‑off. Future work aims to automate data augmentation while preserving fidelity.

Model Evaluation

Current MLLM alignment benchmarks evaluate six key dimensions:

General knowledge : Datasets such as MME‑RealWorld, MMMU, and MMStar assess factual knowledge and multimodal reasoning.

Hallucination : Suites like Object HalBench, VideoHallucer, and VALOR‑Eval detect object‑level, temporal, and relational hallucinations.

Safety : Adversarial attack benchmarks (AdvDiffVLM, RTVLM) and robustness suites (MultiTrust, MOSSBench) measure risk mitigation.

Dialogue : Benchmarks such as Q‑Bench and LLDescribe evaluate conversational competence and visual description quality.

Reward modeling : M‑RewardBench, MJ‑Bench, and MLLM‑as‑a‑Judge assess the quality of learned reward models.

Human‑preference alignment : Arena‑Hard, AlpacaEval‑V2, and MM‑AlignBench quantify correlation with human rankings.

Future Work and Challenges

Key open problems include:

Data scarcity : Fully annotated multimodal datasets larger than 200 k samples are still missing, limiting the diversity of alignment signals.

Visual‑centric alignment : Many current methods rely heavily on textual cues; more effective strategies that directly exploit visual information are needed.

Comprehensive evaluation : Existing studies often evaluate on a narrow set of benchmarks; broader, cross‑task evaluation frameworks are required to assess generalization.

Full‑modality alignment : Early work such as Align‑Anything‑200k demonstrates promise but suffers from limited per‑modality dataset size.

MLLMs as agents : Multi‑agent collaboration, robustness in open environments, and safety mechanisms for embodied systems remain under‑explored.

Insights from Large‑Language‑Model Alignment

Techniques from LLM alignment—data‑efficient fine‑tuning (e.g., LIMA’s 1 k samples), advanced reinforcement‑learning algorithms (PPO variants, sparse‑reward DPO), and multi‑stage collaborative optimization—are guiding the next generation of MLLM alignment research.

Resources

Paper: https://arxiv.org/pdf/2503.14504 GitHub repository:

https://github.com/BradyFU/Awesome-Multimodal-Large-Language-Models/tree/Alignment