LLaDA-Image: Full Diffusion Model Leads Open-Source Text-to-Image via Image-Only Pretraining

Overview

LLaDA-Image, developed by Inclusion AI, is a 6 billion‑parameter Diffusion Transformer (DiT) that adopts a full diffusion architecture: the understanding side uses LLaDA2.0‑mini (a 16B total, 1B active MoE diffusion LLM) and the generation side uses a from‑scratch 6B DiT. The same weights handle both text‑to‑image generation and instruction‑based image editing. On the Qwen‑Image‑Bench benchmark, LLaDA‑Image scores 53.53 (English) and 53.38 (Chinese), ranking first among open‑source models listed in the technical report.

Generated samples — spanning portraits, animals, food, landscapes, posters, and text‑heavy designs — are photorealistic; a visual “real vs. generated” test shows that none of the displayed images are real photographs.

Core Philosophy: Learn to Draw First, Then Learn to Obey

The central innovation is a data‑centric division of labor: pre‑training and mid‑training use pure image supervision (no captions) , while image‑text pairs are introduced only during supervised fine‑tuning (SFT) for language alignment. Across the generation pipeline, over 90% of the cumulative 220 million training samples are image‑only .

Pure images build the visual world; image‑text pairs connect language to that world.

Architecture

The system comprises three modules:

Understanding backbone : LLaDA2.0‑mini (diffusion LLM) processes text, visual conditions, and structured reasoning.

Lightweight interface : a Residual Query Adapter (RQA) extracts generation‑relevant information via cross‑attention, followed by a Transformer Connector that projects hidden states into the DiT’s condition space.

Single‑stream DiT : text condition tokens, timestep embeddings, and image tokens interact in every Transformer layer, predicting a Flow Matching velocity field.

Training Pipeline (Six Stages)

CoT SFT on the understanding backbone.

Image‑only pre‑training at a 256×256 pixel budget.

Image‑only mid‑training at ~ 512×512 with multi‑aspect‑ratio buckets.

Text alignment & high‑resolution transfer at ~ 1024×1024 and ~ 2048×2048 using curated image‑text pairs (captions generated by Qwen3.6‑35B‑A3B and Qwen3‑VL‑235B‑A22B‑Instruct, then verified for objects, attributes, relations, and OCR).

Joint T2I / I2I training (1:1 mix) yielding the unified LLaDA‑Image.

TwinFlow distillation to 2–4 steps, producing LLaDA‑Image‑Turbo.

Image‑Only Pre‑training Mechanics

Each training image serves as both condition and target:

A frozen SigLIP‑VQ encoder converts the image to tokens.

Random masking yields sparse visual tokens; together with a fixed auxiliary instruction they are fed to the frozen multimodal understanding model to form the DiT condition.

The full VAE latents of the unmasked image become the generation target.

This design ensures natural condition‑target alignment, forces context‑based completion, and drives learning of object structure, context relations, and visual composition rules. Only the RQA, connector, and DiT are updated. Because conditions come directly from cropped regions, high‑resolution native crops can be used at low resolution, preserving fine details.

Unified Generation and Editing

Editing reuses the same understanding interface and DiT backbone. The reference image bypasses the language model and supplies two complementary signals:

SigLIP‑VQ features — high‑level semantics identifying “what” is in the reference.

Clean FLUX2‑VAE latents — pixel‑level evidence preserving identity, structure, texture, and background.

Joint 1:1 T2I/I2I training acts as capability replay, maintaining open‑ended generation quality and text alignment while learning editing. Four qualitative editing examples are shown: style transfer to Ghibli, text replacement (“Stay /fresh” → “Keep /fresh”), object removal (red bag), and sketch colorization.

Training Stability Engineering

Long‑run training of a 6B DiT revealed that learnable affine parameters in LayerNorm/RMSNorm cause feature‑scale drift, leading to gradient‑norm explosion. The fix: parameter‑free RMSNorm (no affine weights) throughout the DiT, normalizing by root‑mean‑square and keeping scale fixed. The optimizer is Muon across all generation stages. The schedule is progressive (pre‑train → mid‑train → text alignment → resolution transfer → joint training), with timestep sampling biased toward high noise (where early denoising structure is decided) and checkpoint merging near convergence to smooth mini‑batch variance.

TwinFlow Distillation: One DiT, Two Shifts

TwinFlow builds on Distribution Matching Distillation (DMD) but shares the DiT backbone between the generator and the fake‑score estimator. The trick: positive time steps update the generator, negative time steps update the fake‑score head (1:2 update ratio). At inference, only the generator head remains, so deployment cost equals a single‑head model. LLaDA‑Image‑Turbo (4 steps) scores 50.98 (English) and 50.27 (Chinese) on Qwen‑Image‑Bench, offering a speed‑quality trade‑off.

Benchmark Results

Qwen‑Image‑Bench : English 53.53 / Chinese 53.38 (open‑source #1). Quality, aesthetics, and prompt alignment sub‑scores also lead open‑source models.

LongText‑Bench : English 0.923 / Chinese 0.913.

CVTG‑2K (multi‑text regions): word accuracy 0.875, NED 0.945, CLIPScore 0.818; accuracy holds 0.857–0.892 across 2–5 text regions.

GEdit‑Bench (instruction editing): English 7.336 / Chinese 7.294.

GenEval : 0.85 (single object 1.00, two objects 0.98, attribute binding 0.84, counting 0.53).

DPG‑Bench : 87.48.

Full‑Stack Open Source

The release covers five key assets:

Model weights : base, Turbo, and FP8 variants for 50‑step quality, 2–4‑step speed, and low‑precision deployment.

Training & inference code : core implementation.

Data construction & image‑only training recipes : filtering, 256×256 pre‑training, 512×512 multi‑aspect mid‑training.

Unified generation‑editing recipe : text alignment, high‑res transfer, 1:1 T2I/I2I joint training.

TwinFlow distillation recipe : 50‑step → 2–4‑step path.

Module boundaries (frozen LLaDA2.0‑mini, visual encoder, VAE, RQA, connector) and per‑stage configs are provided, enabling reuse of individual components.

SGLang Day‑0 Support

Inclusion AI collaborated with the SGLang community to provide Day‑0 inference adaptation: few‑step sampling, sequence parallelism, FP8 inference, and an OpenAI Images API‑compatible endpoint for both base and Turbo models.

Conclusion

LLaDA‑Image demonstrates that image‑only pre‑training can establish powerful visual priors , turning raw images from “data waiting for captions” into the primary fuel for visual generation. The open‑sourced pipeline — from visual prior learning through language alignment, unified editing, and few‑step distillation — offers a complete, reproducible research platform.