Multimodal Dialogue Large Model mPLUG-Owl: Technology, Applications, and Evaluation

Multimodal large models have rapidly evolved, with GPT‑4 demonstrating strong vision‑language abilities, prompting the research community to develop open‑source alternatives such as MiniGPT‑4, LLaVA, and Alibaba DAMO Academy's modular multimodal dialogue model mPLUG‑Owl.

The development timeline includes early two‑stage detection‑based methods (UNITER, LXMERT), end‑to‑end approaches (CLIP, ViLT), and recent scaling‑up models that unify image, video, and text tasks, culminating in multimodal dialogue models.

The mPLUG series (E2E‑VLP, mPLUG, mPLUG‑2) introduced a modular training paradigm that achieved state‑of‑the‑art VQA results and laid the foundation for mPLUG‑Owl, which extends the architecture to multimodal dialogue.

mPLUG‑Owl combines a pretrained Vision Transformer visual encoder with large language models (LLaMA, GPT) in a modular design. Training proceeds in two stages: (1) large‑scale image‑text pre‑training to align visual and textual concepts while freezing the language model, and (2) instruction‑tuned fine‑tuning that opens the language side and adds lightweight LoRA adapters.

Evaluation is performed with the OwlEval benchmark, which rates model responses on a four‑level human scale (A–D) across knowledge QA, multi‑turn dialogue, and joke understanding. mPLUG‑Owl consistently receives higher A‑level scores than MiniGPT‑4, LLaVA, and MM‑REACT, demonstrating superior instruction following and fine‑grained visual reasoning.

Beyond QA, mPLUG‑Owl showcases strong OCR, video understanding, and multilingual generation capabilities, handling tasks such as tourism itinerary planning, creative copywriting, document summarization, and video captioning.

The model and its variants are openly released on ModelScope and GitHub, with simple installation instructions and demo interfaces for both English and multilingual usage.

In collaboration with YouKu, the team released the YouKu‑mPLUG video dataset, a large, safety‑filtered Chinese video corpus that provides benchmarks for classification, retrieval, and captioning, aiming to advance Chinese multimodal research.

Overall, mPLUG‑Owl illustrates how modular multimodal pre‑training and instruction tuning can produce a versatile, high‑performance dialogue model that outperforms existing multimodal LLMs across a wide range of applications.