How MetaQuery Bridges MLLMs and Diffusion Models for Superior Multimodal Generation

Background

Unified multimodal models aim to perform both deep understanding (text output) and rich generation (pixel output) within a single architecture. Existing approaches require complex training recipes and careful data balancing to align the two modalities.

Problem

Transferring knowledge from a frozen multimodal large language model (MLLM) to a diffusion generator is difficult because most methods treat the LLM merely as a text encoder, losing in‑context learning and knowledge‑enhanced generation capabilities.

MetaQuery Solution

MetaQuery introduces a set of learnable queries that are fed into a frozen MLLM to extract a detailed visual condition. This condition is passed through a learnable connector (a 24‑layer transformer encoder) and aligned with the input space of a diffusion model. The approach requires only paired image‑caption data and the standard diffusion denoising objective.

Architecture

The model uses LLaVA‑OneVision‑0.5B (or larger Qwen2.5‑VL backbones) as the frozen MLLM and Sana‑0.6B as the diffusion model. Learnable queries have a configurable number N and dimension matching the MLLM hidden size. The connector can be designed as Projection‑Before‑Encoder (Proj‑Enc) or Projection‑After‑Encoder (Enc‑Proj), with Enc‑Proj proving more parameter‑efficient.

Training Strategy

Training proceeds in two stages: (1) pre‑training on 25 M public image‑caption pairs for 8 epochs (learning rate 1e‑4, cosine decay, 4 k‑step warm‑up, batch 4096); (2) instruction‑fine‑tuning on a curated 2.4 M image‑pair dataset built from web corpora using SigLIP clustering and Qwen2.5‑VL‑3B generated instructions. Only the learnable queries, connector, and diffusion model are updated; the MLLM remains frozen.

Key Results

Freezing the MLLM preserves SOTA multimodal understanding while achieving SOTA‑level generation quality.

MetaQuery transfers MLLM knowledge to improve generation reasoning and knowledge‑augmented synthesis.

Learnable queries enable high‑fidelity image reconstruction and editing with minimal fine‑tuning.

Instruction‑tuned MetaQuery exhibits strong zero‑shot subject‑driven generation, logo design, and commonsense reasoning.

Quantitative Evaluation

On MJHQ‑30K, MetaQuery attains a COCO FID of 8.69, surpassing previous Unified Models. It also leads on GenEval, DPG‑Bench, WISE, and CommonsenseT2I benchmarks, demonstrating superior prompt alignment, world‑knowledge reasoning, and visual commonsense performance.

Discussion

Comparisons with different LLM backbones show that instruction‑tuned LLMs improve multimodal understanding without affecting generation. Experiments also reveal that learnable queries outperform the traditional last‑layer embedding approach, especially for knowledge‑enhanced generation.