Multimodal Flow: Fully Continuous Unified Language-Vision Modeling

Background: The Discrete-Continuous Divide in Multimodal Modeling

Building a unified multimodal foundation model that truly integrates language and vision is a key step toward physical AI. Language is inherently a sequence of discrete tokens, while vision consists of continuous high-dimensional pixel distributions. This structural gap has led to two mainstream solutions:

Fully discrete route (e.g., Chameleon, Emu3): Images are quantized into discrete tokens and modeled autoregressively alongside text. However, visual tokenizers lose high-frequency details during quantization.

Hybrid discrete-continuous route (e.g., Transfusion, JanusFlow): Text uses discrete prediction, images use continuous diffusion. This avoids quantization bottlenecks but requires maintaining two distinct objective functions and sampling pipelines, increasing engineering complexity and joint optimization difficulty.

The authors ask whether a third path exists that avoids both compromises.

Multimodal Flow: A Fully Continuous Third Path

Researchers from Huazhong University of Science and Technology (Xinggang Wang's team) and Horizon Robotics propose Multimodal Flow , a fully continuous architecture. In this design, language and vision each remain in their native continuous embedding spaces, yet share a single Flow Matching objective. No forced translation of vision into language tokens, no dual-system maintenance — one Flow model jointly models the combined distribution.

Paper: MULTIMODAL FLOW: UNIFIED FLOW MODELING OF LANGUAGE AND VISION IN EMBEDDING SPACES (arXiv:2609.40362). Code: https://github.com/hustvl/Multimodal-Flow, Project page: https://hustvl.github.io/Multimodal-Flow.

MF-1: 1.6B Parameters, 150B Tokens, Strong Results

The team instantiated the main model MF-1 with only 1.6B parameters, trained from scratch on 150B tokens — a modest budget compared to trillion-token models. Yet results are striking:

Text-to-image generation : GenEval overall score 0.821 , DPG-Bench long-prompt alignment 83.44 . Notably, GenEval Counting dimension reaches 0.73 , far ahead of same-scale unified models.

Multimodal understanding : VQAv2 72.6 , MMBench 67.2 , POPE 86.1 ; average 75.3 , surpassing some same-size models initialized from pretrained LLMs.

Controlled comparison (same data, parameters, optimizer): Multimodal Flow outperforms both Transfusion-style hybrid and Chameleon-style fully discrete architectures across GenEval, GQA, VQAv2, MMBench.

Core Architecture: Hyperchunks and Chunk-Causal Flow Backbone

The key innovation is the Hyperchunk design:

Text is split into chunks of 8 tokens, encoded by a frozen T5-small encoder into continuous text hyperchunks .

Images are encoded by a frozen SigLIP2 encoder into a 16×16 spatial grid, forming a visual hyperchunk .

Hyperchunks are arranged in a task-defined sequence and fed into a shared Chunk-Causal Flow backbone .

The backbone uses a Chunk-Causal attention mask : when predicting the k-th hyperchunk, all previous hyperchunks are fully visible, positions within the current hyperchunk are jointly modeled, and future hyperchunks are masked. This preserves causality while allowing intra-chunk interaction.

A hybrid strategy is employed: attention layers are shared to enable cross-modal interaction, while FFN layers are modality-specific because language and vision representation statistics differ greatly; forced sharing hurts performance (confirmed by ablation).

Training allows parallel prediction of multiple target hyperchunks with independent Flow time-step sampling. Inference proceeds sequentially chunk-by-chunk, reusing KV caches of already-generated hyperchunks for speed.

An elegant consequence: changing hyperchunk order and target direction lets the same parameters seamlessly switch tasks — text generation, text-to-image, image-to-text, VQA — all expressed as conditional generation of hyperchunk sequences. Task definition and generation modeling are fully decoupled.

Mixed Pretraining: Scaling and Transfer Evidence

The team trained 0.6B, 1.2B, and 1.6B models under the same mixed pretraining protocol. Scaling curves show healthy trends: Flow Matching loss and language perplexity decrease steadily, while GenEval (text-to-image) and CLIPScore (image-to-text) rise. The 1.6B model achieves the strongest overall performance late in training.

Transfer learning comparison: random initialization + downstream fine-tuning vs. mixed pretraining + fine-tuning. Gains are substantial:

GenEval: 0.527 → 0.821

SEED-Bench: 31.6 → 62.4

MMBench: 36.0 → 67.2

This demonstrates that the joint representations learned during mixed pretraining under the unified Flow Matching objective transfer efficiently to downstream tasks, establishing Multimodal Flow as a powerful multimodal pretraining foundation.

Key Ablation Findings

1. What Visual Representation Works Best?

The team compared semantic features (SigLIP2, DINOv2), reconstruction-oriented latents (FLUX.2 VAE, SD-VAE), and raw pixels. Under equal training budget, SigLIP2 and DINOv2 outperform VAE latents on generation and captioning tasks. This suggests that for unified generative models, semantically rich features matter more than pixel-level reconstruction fidelity.

2. One CFG Mechanism Works Bidirectionally

Classifier-Free Guidance (CFG) is traditionally used only for visual generation. Thanks to the unified Flow Matching objective, the same CFG mechanism guides both text and image generation . Optimal guidance scale: ~3 for image-to-text, ~5 for text-to-image. This further validates the simplicity and generality of the fully continuous shared generative process.

Outlook

Multimodal Flow and MF-1 demonstrate a viable alternative: vision need not be forced into language tokens, nor do we need two disjoint generation objectives. In continuous embedding spaces, language and vision can be modeled with the same dynamical system while preserving their structural properties. The current 1.6B/150B-token model is just a starting point; the authors plan to extend the framework to longer interleaved image-text sequences, video, and other structured modalities. The imagination space for continuous unified multimodal models may have only just opened.