Why Dropping VAE and Private Data Boosts Text-to-Image Generation Performance

Background

Text-to-image generation has become a highly competitive field, typically relying on a pretrained VAE encoder-decoder, a text encoder, carefully designed conditional injection (e.g., AdaLN), massive datasets, and an RL or DPO alignment stage.

MiniT2I: A Minimalist Baseline

He Kaiming’s team proposes MiniT2I, a deliberately minimalist pixel-space diffusion model that removes the VAE, AdaLN, auxiliary losses, private data, and alignment stages. The model directly denoises RGB pixels using a diffusion objective.

With 258 M parameters (B/16 configuration), MiniT2I reaches 0.87 on GenEval and 84.2 on DPG-Bench, surpassing larger pixel-space baselines.

Design Choices

Pixel-space diffusion without VAE – By operating on 512×512 images split into 16×16 patches (1024 tokens), the sequence length stays within a Transformer’s comfortable range. Removing the VAE cuts per-step FLOPs from ~1379 GFLOPs to ~570 GFLOPs (B/16) and eliminates reconstruction error.

Experiments show comparable FID (18.7 vs 19.0) to latent-space models while reducing step cost by five-fold.

MM-JiT Architecture

Inspired by SD3’s MM-DiT, MiniT2I replaces the AdaLN-based conditioning with two lightweight text-adapter Transformer blocks placed before the joint attention layer, allowing a frozen T5 encoder to adapt to the denoiser’s needs. The AdaLN branch is removed; time-step information is inferred directly from the noisy image.

Removing AdaLN reduces parameters and enables deeper models (12 → 17 layers) under the same compute budget. FID improves from 18.7 to 13.7, and the architecture becomes easier to understand and modify.

Training Data and Procedure

Training follows a two-stage “pre‑train → fine‑tune” paradigm similar to large language models:

Pre‑training on LLaVA-recaptioned CC12M for 250 k steps.

Fine‑tuning on ~120 k high-quality image-text pairs (BLIP3‑o 60 k + LAION DALL·E 3 Discord set + ShareGPT‑4o‑Image) for 40 k steps.

Ablation studies show that both stages are essential: pre‑training alone yields good image quality but poor prompt adherence; fine‑tuning alone leads to narrow visual diversity.

Results

MiniT2I‑B/16 (≈600 M total parameters, including the text encoder) outperforms models with three to four times more parameters on GenEval and DPG‑Bench. Training on eight H100 GPUs for about three days matches the FLOP budget of a standard ImageNet 200‑epoch run.

Scaling to L/16 (912 M parameters) improves style diversity, spatial relations, and text rendering, reaching quality comparable to SD3‑Medium (~2 B parameters). On PRISM‑Bench, MiniT2I‑L/16 scores 79.9 (style), 78.4 (composition), 57.9 (imagination), close to SD3‑Medium, but lags in text rendering (30.6 vs 50.9) and named‑entity generation (60.3 vs 66.3), which the authors attribute to the limits of publicly available data.

Limitations and Outlook

Patch-level artifacts: gradients at patch borders are 17‑22 % higher than interior regions.

High CFG values (~6) cause local tokens to leave the data manifold, exposing visual defects without a decoder’s smoothing.

Resolution ceiling: current 512×512 performance degrades at 4K+ without longer sequences or more efficient attention.

Data bottleneck: text rendering and named‑entity generation remain weaker than industrial systems, requiring specialized data.

MiniT2I demonstrates that strong text-to-image generation does not require massive private datasets or complex conditioning mechanisms, suggesting a paradigm shift from “data‑stacking” to “model‑purification.”

Resources

Paper: “A Minimalist Baseline for Text-to-Image Generation”. Technical blog: https://peppaking8.github.io/#/post/minit2i. Open‑source code: https://github.com/PeppaKing8/minit2i-jax.