Can Lumina-mGPT 2.0 Replace Diffusion Models? A Deep Dive into Its Autoregressive Power

Highlights

Lumina-mGPT 2.0 is a stand‑alone decoder‑only autoregressive model designed to revive the AR paradigm for high‑quality image synthesis.

Its generation quality matches state‑of‑the‑art diffusion models (e.g., DALL·E 3, SANA) while retaining AR flexibility.

A unified tokenization scheme enables a single framework to handle subject‑driven generation, image editing, controllable synthesis, and dense prediction.

Efficient decoding strategies improve both quality and speed.

The model is fully open‑source and free of external pretrained weights.

Problem Statement

Marginalisation of AR in image generation – despite early success, AR models have been eclipsed by diffusion and GANs.

Dependence on pretrained components – many multimodal generators rely on external visual encoders or hybrid diffusion‑AR pipelines, leading to architectural fragmentation and licensing constraints.

Lack of unified generation capability – existing models often cannot handle generation, editing, and controlled synthesis within a single system.

Resource‑efficiency vs. flexibility trade‑off – pure AR models such as Emu3 are simple but suffer from low quality and high compute cost.

Proposed Solution

Introduce Lumina-mGPT 2.0, a pure decoder‑only AR model trained from scratch (no pretrained weights).

Adopt a unified tokenization that encodes images and text into the same token stream, supporting multiple tasks (subject‑driven generation, editing, controllable synthesis, dense prediction).

Integrate efficient inference strategies:

Inference‑time scaling to boost quality.

Speculative Jacobi sampling (SJD) to accelerate decoding.

Technical Foundations

Pure AR Architecture – a decoder‑only Transformer identical to large language models, predicting the next token for both image and text.

Multimodal Unified Tokens – image and text are both discretised, removing the need for separate pretrained encoders.

Training Strategy – three‑stage progressive resolution fine‑tuning (FP‑SFP) with flexible‑progressive supervised fine‑tuning; training runs on 64 A100 GPUs for 4–5 weeks.

Model Sizes – 2 B and 7 B parameter variants; larger models converge faster and produce higher‑fidelity images.

Inference Optimisations

Model quantisation to 4‑bit integers (group size 128) while keeping activations in bfloat16, applied via torch.compile reduce‑overhead mode.

Static KV‑cache and static causal‑mask design to make SJD compatible with compiled kernels.

Experiments

Training Data – a filtered subset of Lumina‑Image 2.0 containing real and synthetic images, plus task‑specific datasets (Subject200K, OmniEdit, etc.).

Evaluation Benchmarks – GenEval, DPG for text‑to‑image alignment; VisualCloze for controllable and subject‑driven generation.

Quantitative Results

On GenEval and DPG, Lumina‑mGPT 2.0 matches or exceeds diffusion baselines, achieving top‑tier scores (exact numbers omitted for brevity). It also outperforms prior AR models such as Emu3 and Janus Pro.

In controllable generation (Canny, Depth) and subject‑driven tasks, the model shows superior structural consistency and image‑text alignment.

Qualitative Results

Generated samples demonstrate realistic humans, vivid landscapes, sci‑fi scenes, and detailed close‑ups, confirming strong prompt understanding.

Ablation Studies

Model scaling – the 7 B version consistently beats the 2 B version across all metrics.

Pre‑thinking prompts – using GPT‑4o to refine user prompts improves GenEval scores on object count, position, and colour attributes by noticeable margins.

Inference‑time scaling – sampling multiple images and selecting the best improves accuracy on multi‑object sub‑tasks.

Sampling acceleration – quantisation reduces sampling time by X % and memory usage by Y %; adding SJD further cuts time by Z % without quality loss.

Limitations

Sampling still takes minutes per image, a common bottleneck for AR generators. The “pre‑thinking” step relies on an external LLM (GPT‑4o); future work aims for self‑contained reasoning. Current focus is multimodal generation; broader multimodal understanding is left for later versions.

Conclusion

Lumina‑mGPT 2.0 proves that a decoder‑only, zero‑shot trained autoregressive model can rival diffusion systems in image quality while offering a unified, licence‑free platform for diverse visual tasks. Nevertheless, inference speed and external LLM dependence remain open challenges.

References

[1] Lumina‑mGPT 2.0: Stand‑Alone Autoregressive Image Modeling