Skip to content
Multimodal Flow: A Fully Continuous Third Path for Unified Language-Vision Modeling

Multimodal Flow: A Fully Continuous Third Path for Unified Language-Vision Modeling

HUST and Horizon Robotics propose Multimodal Flow, a fully continuous flow-matching architecture that unifies language and vision in embedding spaces without tokenization or dual objectives, achieving state-of-the-art results with a 1.6B model trained on only 150B tokens.

Data Party THU
Data Party THU
Data Party THU

Researchers from Huazhong University of Science and Technology (HUST) and Horizon Robotics introduce Multimodal Flow , a novel fully continuous approach to unified multimodal modeling that avoids both the quantization loss of fully discrete methods (e.g., Chameleon, Emu3) and the engineering complexity of hybrid discrete-continuous systems (e.g., Transfusion, JanusFlow). The paper, "MULTIMODAL FLOW: UNIFIED FLOW MODELING OF LANGUAGE AND VISION IN EMBEDDING SPACES," is available on arXiv (https://arxiv.org/pdf/2609.40362) with code at https://github.com/hustvl/Multimodal-Flow and project page at https://hustvl.github.io/Multimodal-Flow.

Core Architecture: Hyperchunk + Chunk-Causal Flow

The key innovation is the Hyperchunk design. Text is segmented into chunks of 8 tokens each, encoded by a frozen T5-small encoder into continuous text hyperchunks. Images are processed by a frozen SigLIP2 encoder into 16×16 spatial grid features, forming a single visual hyperchunk. These hyperchunks are arranged in a task-defined sequence and fed into a shared Chunk-Causal Flow backbone.

The backbone employs a Chunk-Causal attention mask : when predicting the k-th hyperchunk, all previous hyperchunks are fully visible, positions within the current hyperchunk are jointly modeled, and future hyperchunks are masked. This preserves causality while allowing rich intra-chunk interaction. The network uses a joint attention + modality-specific FFN strategy: attention layers are shared to enable cross-modal interaction, while FFN layers remain separate because language and vision representations have vastly different statistics (ablation confirms forced sharing hurts performance).

Training allows parallel prediction of multiple target hyperchunks with independent flow time-step sampling per chunk. Inference proceeds sequentially chunk-by-chunk, reusing KV caches of already-generated hyperchunks for efficiency. Crucially, changing hyperchunk order and conditioning direction lets the same parameters seamlessly switch tasks — text generation, text-to-image, image-to-text, VQA — all expressed as conditional hyperchunk sequence generation.

Experimental Results: MF-1 (1.6B Parameters, 150B Tokens)

The team instantiated MF-1, a 1.6B parameter model trained from scratch on only 150B tokens — modest by current standards. Results:

Text-to-image : GenEval composite score 0.821, DPG-Bench long-prompt alignment 83.44. Notably, GenEval Counting dimension reaches 0.73, far ahead of same-scale unified models.

Multimodal understanding : VQAv2 72.6, MMBench 67.2, POPE 86.1, average 75.3 — surpassing some same-size models initialized from pretrained LLMs.

Controlled comparison (same data, parameters, optimizer): Multimodal Flow outperforms both Transfusion-style hybrid and Chameleon-style fully discrete architectures across GenEval, GQA, VQAv2, and MMBench.

Scaling and Transfer Evidence

Three model sizes (0.6B, 1.2B, 1.6B) trained under identical mixed-pretraining protocol show healthy scaling: Flow Matching loss and language perplexity decrease steadily, while GenEval (text-to-image) and CLIPScore (image-to-text) rise consistently, with the 1.6B model achieving strongest overall performance at late training stages.

Transfer learning comparison: Mixed pretraining + fine-tuning vs. random initialization + fine-tuning yields dramatic gains — GenEval 0.527 → 0.821, SEED-Bench 31.6 → 62.4, MMBench 36.0 → 67.2 — confirming that joint representations learned under the unified Flow Matching objective transfer effectively to downstream tasks.

Key Ablation Findings

Visual representation matters : Comparing SigLIP2, DINOv2 (semantic features) against FLUX.2 VAE, SD-VAE (reconstruction latents) and raw pixels, semantic features consistently outperform on generation and captioning under equal compute. This suggests semantic richness outweighs pixel-level reconstruction fidelity for unified generative modeling.

Single CFG works bidirectionally : Classifier-Free Guidance, traditionally used only for visual generation, applies to both modalities under the unified Flow Matching objective. Optimal guidance scale is ~3 for image-to-text and ~5 for text-to-image, demonstrating the elegance of a shared generative process.

Future Directions

The authors position the 1.6B/150B-token model as a starting point. Next steps include scaling to longer interleaved image-text sequences, video, and other structured modalities, suggesting the design space for continuous unified multimodal models is just opening.

Code example

来源:机器之心
本文
约3000字
,建议阅读
5
分钟
如何在一个模型内统一这两种截然不同的分布,一直是一个备受争论的话题。
Original Source

Signed-in readers can open the original source through BestHub's protected redirect.

Sign in to view source
Republication Notice

This article has been distilled and summarized from source material, then republished for learning and reference. If you believe it infringes your rights, please contactadmin@besthub.devand we will review it promptly.

Flow MatchingVision-Language ModelsGenerative AIMachine Learning ResearchHyperchunkMultimodal FlowContinuous RepresentationsUnified Multimodal Modeling
Reader feedback

How this landed with the community

Sign in to like

Rate this article

Was this worth your time?

Sign in to rate
Discussion

0 Comments

Thoughtful readers leave field notes, pushback, and hard-won operational detail here.