Why ByteDance’s 7B BAGEL Model Rivals GPT‑4o in Unified Multimodal Understanding and Generation

Overview

ByteDance released the open‑source 7B‑parameter unified multimodal model BAGEL (paper: Emerging Properties in Unified Multimodal Pretraining , arXiv 2505.14683). Trained on a massive interleaved multimodal dataset (text, images, video, web data) using a Mixture‑of‑Transformers (MoT) backbone, BAGEL demonstrates strong understanding, generation, and editing abilities, even showing emergent behaviours comparable to GPT‑4o.

Model Architecture

BAGEL adopts a Transformer‑Experts (MoT) design with two experts: an understanding expert for text and ViT tokens, and a generation expert for VAE tokens. Both experts share a multimodal self‑attention layer that processes an interleaved token sequence. The backbone is a decoder‑only LLM initialized from Qwen2.5, enhanced with RMSNorm, SwiGLU, RoPE, and GQA. Two visual encoders are used:

Understanding encoder: ViT initialized from SigLIP2‑so400m/14, with 384‑pixel resolution, NaViT for native aspect‑ratio handling, and an MLP connector to match LLM hidden size.

Generation encoder: Pre‑trained VAE from FLUX, frozen during training, converting pixels to latent space (down‑sample ×8, 16 channels) and projecting to LLM dimensions.

Position encodings are added to ViT and VAE tokens before integration. Diffusion timestep embeddings follow the CausalFusion approach, avoiding AdaLN.

Training Data

The dataset combines pure text, image‑text pairs, and novel interleaved multimodal streams. Video data (Koala36M + MVImgNet2.0) provides temporal and physical continuity, while web data (OmniCorpus) supplies diverse interleaved text‑image resources. In total, BAGEL uses 45 M interleaved image‑text samples and 20 M interleaved video‑text samples, plus 500 k reasoning‑enhanced examples.

Training Strategy

BAGEL follows a four‑stage schedule:

Alignment: Freeze LLM and vision encoder, train only the MLP connector on 378×378 image‑text pairs.

Pre‑training: Open all parameters (except VAE) on 2.5 T tokens covering text, image‑text, multimodal conversation, web‑interleaved, and video‑interleaved data.

Continued Training: Increase visual resolution and interleaved data sampling, consuming an additional ~2.6 T tokens.

Supervised Fine‑tuning: Curate high‑quality subsets for generation (image‑text & interleaved generation) and understanding (LlaVA‑OV & Mammoth‑VL), training on 72.7 B tokens.

Experiments show that raising the generation‑data sampling ratio from 50 % to 80 % steadily reduces MSE loss, while learning‑rate adjustments trade off CE loss versus MSE loss, prompting separate weighting for the two objectives.

Emergent Abilities

Training curves reveal a phase‑transition pattern: basic understanding and generation converge early (≈0.18 T and 0.68 T tokens), followed by simple editing (≈2.64 T tokens) and finally intelligent editing (≈3.61 T tokens). Qualitative case studies illustrate that after ~3.5 T tokens the model begins to perform coherent, multi‑step visual reasoning rather than merely copying inputs.

Benchmark Results

Visual Understanding: On MMMU and MM‑Vet, BAGEL (7B active parameters) outperforms Janus‑Pro by 14.3 and 17.1 points respectively, and exceeds specialized models such as Qwen2.5‑VL and InternVL2.5 on most metrics.

Visual Generation: BAGEL achieves an 88 % overall score on GenEval, surpassing FLUX‑1‑dev (82 %) and SD3‑Medium (74 %). On the WISE benchmark it beats all open models and only lags behind proprietary GPT‑4o.

Image Editing: On GEdit‑Bench, BAGEL matches the leading Step1X‑Edit and beats Gemini 2.0. On the newly introduced IntelligentBench it scores 44.9, well above Step1X‑Edit (30). Qualitative comparisons (Figures 17‑21) show BAGEL producing higher‑quality edits and fewer unintended modifications than GPT‑4o.

Chain‑of‑Thought (CoT) Enhancements: Adding a CoT reasoning step before generation raises WISE scores from 0.52 to 0.70 (+0.18) and improves IntelligentBench from 44.9 to 55.3, demonstrating the benefit of explicit reasoning.

Failure Cases: BAGEL struggles with IP‑protected content, complex textual prompts, counterfactual scenarios, object swaps, and de‑blurring tasks, where GPT‑4o still leads.

Key Takeaways

BAGEL proves that a relatively small (7 B) unified multimodal model, when trained on high‑quality interleaved data and a MoT architecture, can achieve state‑of‑the‑art performance across understanding, generation, and editing, and exhibit emergent capabilities comparable to much larger proprietary systems.