Progress in Multimodal Large Language Models: Background, Architecture, Evolution, Team Work, and Future Outlook

With the rapid development of AI, multimodal large language models (MLLMs) have become a hot research topic. Traditional text‑only LLMs are limited to textual input and output, while real‑world information is often visual or auditory, prompting the emergence of multimodal models that can process images, video, and audio.

Recent industry efforts include OpenAI's GPT‑4V/4o, Google's Gemini Pro, and domestic projects such as Alibaba Cloud's Qwen VL. Since late 2022, over a hundred new multimodal models have been released in the open‑source community.

Typical application cases demonstrated are image captioning, counting objects in images, object localization with bounding boxes, complex visual reasoning (e.g., chart inference and code generation from images), and multi‑image or video understanding.

The prevailing architecture of MLLMs consists of three parts: (1) an encoder that converts raw visual signals into high‑level tokens (e.g., CLIP producing 256 visual tokens), (2) a connector that aligns visual and textual representations—either a simple MLP that projects and concatenates tokens or a Q‑former that compresses visual tokens via learnable queries, and (3) a pretrained large language model that provides the reasoning and generation capabilities.

Training is usually divided into two stages: modal alignment using image‑caption pairs to teach the model visual semantics, followed by instruction fine‑tuning on diverse tasks (visual QA, detection, etc.) to enable the model to follow new instructions and generalize to unseen tasks.

Evaluation methods include conventional task‑specific test sets (e.g., VQA) and advanced capability benchmarks that assess complex reasoning, common‑sense understanding, and code inference. Results show that closed‑source models still outperform open‑source ones, especially on coarse perception tasks, while fine‑grained counting remains challenging.

The presenting team’s recent work focuses on reducing hallucinations by integrating expert perception models (object detectors, VQA) with LLMs in a plug‑and‑play, training‑free framework, achieving notable improvements on models like mPLUG. They also built a long‑video understanding benchmark with 900 manually annotated videos and 2,700 QA pairs, revealing that current models struggle with fine‑grained perception in long videos.

Future research directions highlighted are extending multimodal context length (handling more visual tokens or higher‑resolution inputs), developing embodied agents for on‑device assistance, and creating unified multimodal models that can both understand and generate images, enabling more natural human‑AI interaction.

Overall, the article provides a comprehensive overview of the state‑of‑the‑art in multimodal LLMs, practical challenges, and promising avenues for further investigation.