How Multimodal Alignment Is Shaping the Future of Large Language Models

Overview

This review surveys recent progress on alignment algorithms for multimodal large language models (MLLMs). It categorises the work by application scenario, data construction, evaluation methodology, and future research directions, highlighting the technical details needed to reproduce or extend the methods.

Application Scenarios and Representative Alignment Methods

General Image Understanding

The primary goal is to reduce hallucinations while improving dialogue and reasoning. Early work such as Fact‑RLHF introduced a multimodal RLHF pipeline with 10K human‑annotated preference pairs, applying per‑token KL penalties, factuality calibration, and length penalties. Subsequent improvements include:

DDPO : re‑weights correction data to strengthen DPO training.

HA‑DPO : generates image captions with an MLLM, validates them with GPT‑4, and rewrites positive/negative samples, adding a causal language modelling loss to suppress hallucinations.

mDPO : incorporates a visual loss term and an anchoring mechanism to prevent probability collapse of preferred responses.

Beyond hallucination mitigation, methods such as Silkie collect diverse instruction data and use GPT‑4V to score responses for DPO, while CLIP‑DPO leverages CLIP similarity scores as rewards, simultaneously improving zero‑shot classification. SIMA lets the model self‑evaluate its outputs to generate preference pairs, and MM‑RLHF expands data diversity to further boost alignment performance.

Multi‑Image, Video, and Audio

Multi‑image : MIA‑DPO builds multi‑image preference datasets, enabling the model to reason over several images jointly.

Video : LLaVA‑NeXT‑Interleave combines DPO with interleaved visual instruction tokens to handle temporal information.

Audio‑Visual : Video‑SALMONN 2 introduces an audio‑visual alignment module that resolves the “audio‑blindness” problem by jointly encoding spectrograms and video frames.

Domain‑Specific Extensions

Medical Imaging : 3D‑CT‑GPT++ fine‑tunes on CT scans with alignment losses, achieving clinical‑grade diagnostic accuracy.

Mathematical Reasoning : MAVIS redesigns the visual‑math pipeline, improving performance on benchmarks such as MathVista.

Safety : AdPO and VLGuard construct adversarial image‑text pairs and apply robust training to mitigate attacks.

Agent‑Level Reasoning : INTERACTIVECOT and EMMOE dynamically decompose tasks and optimise reasoning pipelines for complex decision‑making.

Alignment Data Construction

Datasets are divided into two broad families.

External‑knowledge datasets : Collected via human annotation or generated by closed‑source models (e.g., GPT‑4). Examples include LLaVA‑RLHF (10K human‑selected pairs) and RLHF‑V (1.4K corrected hallucination samples). These provide high quality but are expensive.

Self‑annotated datasets : Generated from the model itself. Sub‑categories are:

Single‑text modality : SQuBa creates negative samples with a fine‑tuned model; SymDPO converts VQA data to in‑context learning format.

Single‑image modality : Image DPO perturbs images (Gaussian blur, pixelation) while keeping text unchanged to form preference pairs.

Image‑text mixed modality : AdPO builds original vs. adversarial image‑text pairs, using differing visual and textual content for positive/negative samples.

Key trade‑offs are scale versus quality and the need for automated data‑augmentation to increase diversity without sacrificing reliability.

Model Evaluation Benchmarks

Current MLLM alignment evaluation is organised along six dimensions.

General Knowledge : Datasets such as MME‑RealWorld (13K images, 29K Q‑A pairs) and MMMU (11.5K academic questions) test factual and reasoning abilities.

Hallucination : Benchmarks like Object HalBench , VideoHallucer , and HallusionBench quantify object‑level, temporal, and relational hallucinations using human annotations and synthetic data.

Safety : VLGuard , AdvDiffVLM , and RTVLM evaluate robustness against adversarial attacks, red‑team scenarios, and out‑of‑distribution inputs.

Dialogue : Suites such as Q‑Bench , LLDescribe , and LLaVA‑Bench‑Wilder assess conversational consistency and visual description quality.

Reward Model Performance : Benchmarks like M‑RewardBench (23 languages) and MM‑RLHF‑RewardBench measure the fidelity of learned reward models.

Alignment with Human Preferences : Arena‑Hard (98.6% correlation with human rankings) and MM‑AlignBench provide hand‑annotated human‑preference scores.

Each dimension contains multiple datasets that together form a comprehensive evaluation framework for hallucination reduction, safety, reasoning, and preference alignment.

Future Work and Open Challenges

Data Scarcity : High‑quality multimodal alignment data remain limited (<200K fully human‑annotated samples). Scaling requires cost‑effective self‑annotation and automated augmentation.

Visual Information Utilisation : Many methods rely on textual cues for positive/negative sampling. Approaches such as using corrupted images, generating new questions from altered images, or CLIP‑based cosine similarity each have trade‑offs in computational cost and bias.

Comprehensive Evaluation : Existing studies focus on narrow benchmarks. A unified cross‑task suite covering general knowledge, hallucination, safety, dialogue, reward modelling, and human‑preference alignment is needed.

Full‑Modality Alignment : Align‑Anything introduced a 200K multimodal dataset spanning text, image, audio, and video, demonstrating complementary gains across modalities. However, each modality’s dataset is still relatively small, and the alignment algorithm (a DPO variant) does not fully exploit modality‑specific structures.

Training Efficiency : Standard DPO requires loading both policy and reference models, slowing training. Reference‑free methods such as SimPO promise faster convergence and lower memory usage.

Over‑Optimization / Reward Gaming : Balancing data diversity, early stopping, and regularisation is essential to prevent reward‑model over‑fitting.

Insights from Large Language Model Alignment

LLM alignment research offers two practical lessons for MLLMs:

Improving training efficiency by reducing dependence on a reference model (e.g., SimPO, GRPO).

Mitigating reward‑gaming through balanced datasets, early‑stop criteria, and regularisation techniques.

MLLMs as Autonomous Agents

To transform MLLMs into effective agents, three research fronts must be addressed:

Multi‑agent Collaboration : Existing multi‑agent frameworks are text‑centric; extending them to multimodal agents requires new communication protocols and shared perception modules.

Robustness in Open Environments : Systematic adversarial testing (e.g., AdvDiffVLM) and safety‑oriented fine‑tuning (VLGuard) are prerequisites for deployment.

Safety Guarantees : Incorporating safety‑aware alignment losses and red‑team evaluations reduces the risk of harmful outputs when agents interact with real‑world data streams.

Overall, the surveyed methods illustrate a rapid evolution from hallucination‑focused alignment toward holistic, multimodal, and agent‑oriented systems. Continued progress will depend on larger, higher‑quality datasets, more efficient training paradigms, and unified evaluation protocols that span the full spectrum of multimodal capabilities.

Code example

相关阅读：