How BitDance’s 2.6B‑Parameter Model Beats 14B Counterparts with 8.7× Speedup

Binary Visual Tokenization

BitDance et al. introduce a binary visual tokenizer with a vocabulary size of 2^256, avoiding the collapse of traditional vector‑quantized tokenizers.

The tokenizer outputs 256‑bit tokens, each representing a vertex of a high‑dimensional hypercube.

Binary Diffusion Head

A binary diffusion head jointly models all 256 bit channels using a rectified‑flow formulation. Continuous diffusion predictions are projected onto hypercube vertices, enabling precise sampling in the astronomically large discrete space without requiring a trillion‑parameter classification head.

Next‑Block Diffusion for Parallel Autoregressive Generation

Tokens are partitioned into non‑overlapping blocks. A block‑wise causal attention mask allows tokens within the same block to attend to each other while preserving autoregressive dependencies across blocks. The binary diffusion head works with a lightweight Diffusion Transformer (DiT) to predict all tokens in a block simultaneously.

Training and Inference Details

Model variants: BitDance‑B‑4x (2.6 B parameters) and BitDance‑H‑1x (1 B parameters).

Training stages: pre‑training, continual training, supervised fine‑tuning, followed by knowledge distillation.

Mixed‑resolution training uses a 512‑pixel base resolution with additional 256‑ and 1024‑pixel samples.

Knowledge distillation reduces inference time to 12.4 seconds for a 1024×1024 image (≈30× speed‑up over token‑by‑token baselines).

Benchmark Results

ImageNet conditional generation (32× down‑sampling): PSNR = 25.29 for BitDance‑B‑4x, surpassing continuous VAE baselines.

FID: BitDance‑H‑1x achieves 1.24 without extra self‑supervised models; BitDance‑B‑4x outperforms RandAR‑XXL by ~0.5 FID points.

Text‑to‑image: uses pretrained Qwen‑3‑14B as both text encoder and image generator, with a 16× down‑sampled visual tokenizer and 16‑block parallel generation.

Additional metrics: GenEval = 0.86, DPG‑Bench = 88.28.

Ablation Studies

Replacing the binary diffusion head with per‑bit classification dramatically drops PSNR, FID and other metrics, confirming the importance of modeling bit‑wise dependencies.

Substituting the binary tokenizer with a continuous VAE also degrades performance, highlighting risks of unconstrained continuous tokens.

Resources

Code and models: https://github.com/shallowdream204/BitDance

Paper: https://arxiv.org/pdf/2602.14041

Project site: https://bitdance.csuhan.com/