PRA Beats 1.9B Baseline Without a Visual Tokenizer: 135M Model Surpasses Large-Scale Pixel‑Space AR

Motivation

Direct pixel‑space autoregressive (AR) image generation avoids visual tokenizers and the associated compression loss, but historically suffers from poor generation quality. The difficulty stems from two aspects: (1) each pixel patch is both the output to be predicted and the input context for subsequent steps, and (2) errors in early patches can propagate through the autoregressive chain.

Diagnostic Experiments

To isolate the output bottleneck, the authors trained AR and diffusion (JiT) models under identical settings while varying the token dimensionality. With a low token dimension (48), AR and diffusion performance were comparable. When the dimension increased to 768, AR’s FID degraded sharply, revealing that predicting high‑dimensional continuous pixel tokens in a single step is substantially harder than diffusion’s multi‑step refinement.

To examine the input bottleneck, they compared teacher‑forced training (clean ground‑truth prefixes) with inference (model‑generated prefixes). Injecting Gaussian noise into input tokens during training improved AR performance, indicating that exposing the model to imperfect inputs is beneficial. However, simple noise injection did not fully emulate the distribution of tokens generated during rollout, and true on‑policy rollout was deemed too costly for efficient training.

Parallel Rollout Approximation (PRA)

Output modification: PRA replaces the direct prediction of a 768‑dimensional pixel patch with the prediction of a learned 16‑dimensional intermediate state. A pixel decoder then maps this intermediate state back to the pixel patch, dramatically reducing per‑step prediction difficulty. The intermediate state is learned jointly with the AR transformer and is not derived from an external tokenizer.

Input modification: During training, PRA adds noise to the intermediate state, decodes it to pixel inputs, and feeds these decoded pixels to the AR transformer. This parallel construction approximates the distribution of rollout inputs while preserving the parallelism of teacher‑forced training, thereby narrowing the training‑inference distribution gap.

Experimental Setup

Evaluations were performed on ImageNet‑1K 256×256 class‑conditional generation. Images were split into 16×16 patches (sequence length 256, token dimension 768). Three model scales were tested: PRA‑Small (135 M parameters), PRA‑Base (250 M), and PRA‑Large (511 M).

Results

PRA‑Small achieved FID 2.58, outperforming the 1.9 B‑parameter FARMER‑1.9B/8 baseline (FID 3.60) with roughly one‑fourteenth the parameters.

PRA‑Base reached FID 2.21.

PRA‑Large attained FID 1.94, establishing a new state‑of‑the‑art for pixel‑space AR.

Linear probing on ImageNet showed that PRA‑Large obtained a top‑1 accuracy of 68.80%, surpassing existing AR and diffusion baselines and demonstrating strong visual representation learning.

Conclusion

By jointly addressing high‑dimensional token prediction and the training‑inference distribution mismatch, PRA enables high‑quality pixel‑space autoregressive generation without external tokenizers, delivering both superior generative performance and useful visual representations.

Paper: https://arxiv.org/abs/2606.27978

Code: https://github.com/MangataX/PRA