How a 4B-Parameter Open-Source Model Outperforms 14B Multimodal Giants

Overview

InternVL‑U is a 4 B‑parameter unified multimodal model (2 B MLLM + 1.7 B visual generation head) built on the open‑source InternVL‑3.5 backbone. The code is hosted at https://github.com/OpenGVLab/InternVL-U and model weights at https://huggingface.co/InternVL-U/InternVL-U.

Design Principles

Unified Contextual Modeling : a shared latent space for multimodal context tokens, processed with causal masking.

Decoupled Visual Representations : high‑level semantic features from a pretrained ViT are used for understanding, while a VAE‑compressed latent space feeds the generation head.

Modality‑Specific Modularity : separate stems and heads for language and image avoid unnecessary FLOPs.

Core Architecture

The model uses a dual‑stream MMDiT block with gated attention. Visual tokens and generation targets are projected into a common conditional space; an element‑wise gating mechanism mitigates the “attention‑sink” problem in high‑resolution, long‑context scenarios. Position encoding follows a unified 3‑D MSRoPE scheme (time, height, width) with resolution interpolation to support arbitrary up‑sampling.

Training Objectives

Two losses are combined: L = λ_{text}·L_{NTP} + λ_{img}·L_{FM} L_{NTP} is the next‑token prediction cross‑entropy for discrete text tokens. L_{FM} is a flow‑matching loss that regresses the velocity field of the VAE latent trajectory to a linear interpolation between noise and data:

L_{FM}=\mathbb{E}_{t,\mathbf{z}_t}\big\|v_θ(\mathbf{z}_t,t)-v_{true}(t)\big\|^2

Scalar weights λ_{text} and λ_{img} are dynamically adjusted across training stages.

Three‑Stage Progressive Training

Stage 1 – Generation head pre‑training : Freeze the MLLM, train the visual head and projection layers on mixed text‑to‑image and editing data at a fixed 512 px resolution.

Stage 2 – Arbitrary‑resolution continual pre‑training : Keep the MLLM frozen, introduce variable aspect ratios (0.5–2.0) and resolutions (512–1024 px), and inject VAE latent conditions to improve pixel‑level consistency.

Stage 3 – Unified supervised fine‑tuning : Unfreeze the entire model, fine‑tune end‑to‑end with a weighted combination of L_{NTP} and L_{FM}, and add Chain‑of‑Thought (CoT)‑generated reasoning steps to the training data.

Reasoning‑Centric Data Synthesis

The synthetic pipeline creates high‑semantic‑density data in four vertical domains:

Text‑centric : bilingual (Chinese‑English) text rendering and editing pipelines, including OCR‑based text replacement.

Science‑centric : programmatic generation of physics, chemistry, biology, and CS diagrams via GeoGebra, SVG, and matplotlib.

Spatial‑centric : 3‑D geometric transformations, multi‑view CAD, and rotation/translation data.

Humor‑centric (memes) : five‑stage pipeline (text detection → removal → instruction generation) for meme creation.

For each abstract user command (e.g., “draw a weekend meme”), a large language model expands it into explicit CoT steps that enumerate objects, attributes, constraints, and background. This yields richer supervision for the visual head.

Experimental Evaluation

Benchmarks cover understanding, reasoning, generation, and editing.

Model size advantage : 4 B parameters outperform 14–20 B unified baselines (e.g., BAGEL) on almost all tasks.

Understanding & reasoning : Scores on MME‑P, OCRBench, and MMMU match or exceed 14 B BAGEL, with no catastrophic forgetting after adding generation capability.

Text‑to‑image generation : Highest scores on GenEval and DPG‑Bench for object composition and attribute binding; state‑of‑the‑art on CVTG‑2k and LongText‑Bench for bilingual text rendering; large gains on knowledge‑intensive benchmarks (WISE, GenExam) thanks to CoT.

Image editing : Competitive F1 on the TextEdit benchmark (on par with GPT‑Image‑1.5 and Nano Banana Pro); on RISEBench, CoT‑driven editing improves the score from 3.6 to 9.4, surpassing all open‑source and specialized models.

Key Findings

Decoupling visual representations allows the model to retain high‑level semantic reasoning while achieving high‑fidelity synthesis.

Gated attention in the dual‑stream MMDiT block effectively prevents attention‑sink degradation in long‑context, high‑resolution settings.

Resolution‑interpolated MSRoPE maintains consistent spatial encoding across low‑ and high‑resolution training phases.

CoT‑guided data synthesis bridges the gap between abstract user intent and concrete visual execution, yielding measurable performance jumps on reasoning‑heavy editing tasks.

Conclusion

InternVL‑U demonstrates that a carefully modularized 4 B‑parameter architecture can unify multimodal understanding, reasoning, generation, and editing without sacrificing any individual capability. The combination of unified context modeling, modality‑specific modularity, decoupled visual representations, and a reasoning‑centric data pipeline provides a strong baseline for future research on general‑purpose unified multimodal models.