NVIDIA's PixelUMM: Encoder-Free Unified Image & Video Understanding in Pixel Space

Introduction

Unified multimodal models (UMMs) typically employ separate visual representations for understanding and generation: Vision Transformers (ViT) extract semantic features for understanding, while Variational Autoencoders (VAE) provide latent variables for generation. This dual-token approach lengthens the visual context and diverges from the single visual stream used in vision-language models (VLMs), making it difficult to reuse mature VLM pretraining pipelines.

Recent pixel-space modeling (e.g., JiT) demonstrates that image generation can operate directly on pixels without a VAE. However, extending this paradigm to video is non-trivial: video generation often relies on 3D VAEs, while video understanding typically processes frames independently with image encoders, resulting in mismatched temporal representations. Designing a shared video interface for both understanding and generation remains an open problem.

To address this, NVIDIA and the University of Waterloo propose PixelUMM , an encoder-free model that unifies image and video understanding and generation directly in pixel space.

Model Architecture

Native Pixel Interface

Images are divided into non-overlapping p×p patches ( p=16), and videos into τ×p×p tubelets ( τ=4). Each image patch flattens to 16×16×3 = 768 values; each video tubelet to 4×16×16×3 = 3072 values. A single linear layer maps these to the Transformer hidden dimension.

Separate linear projections are used for understanding vs. generation ( img_und_linear_proj / img_gen_linear_proj, video_und_linear_proj / video_gen_linear_proj): clean images/videos enter via understanding projections, noisy generation targets via generation projections.

Output heads are simple linear layers implemented as RMSNorm plus a reverse linear projection, producing 16×16×3 image patches or 4×16×16×3 video tubelets, which are then unpatchified back to pixels. Output projections are zero-initialized before multimodal training. Consequently, PixelUMM has no visual encoder, no VAE, and no discrete visual tokenizer—only a single linear projection and deterministic patchify/unpatchify operations.

No Explicit Timestep Conditioning

PixelUMM removes explicit timestep embeddings and timestep-conditioned AdaLN. The denoising network receives only noisy pixels and must infer noise intensity from the pixels themselves.

Mixture-of-Transformers Backbone

Initialized from Qwen3's decoder-only Transformer, PixelUMM employs token-level hard routing between an understanding expert and a generation expert. Each block contains separately routed normalization layers, QKV/output projections, and FFNs, while all tokens participate in a shared multimodal self-attention. Text and clean visual tokens route to the understanding expert; noisy visual tokens route to the generation expert.

Unified Multimodal Sequence

Four core tasks are serialized into ChatML conversation format, with visual inputs and generation targets occupying raw pixel token spans: image understanding (I2T) and video understanding (V2T) use clean visual inputs; text-to-image (T2I) and text-to-video (T2V) use noisy targets. Video understanding includes timestamps indicating each video chunk's temporal position.

Generalized Causal Attention

Text uses causal attention; each image or video block uses bidirectional attention; later blocks attend to all earlier blocks. During generation, noisy targets see the prompt and the entire target block, while preceding context cannot see the target. Unlike BAGEL, the sequence contains only text tokens and raw pixel tokens—no VAE or ViT features.

Training Objectives

Text follows BAGEL's design: cross-entropy loss computed only on LLM responses. Pixels follow JiT's design: the model predicts clean pixels (x-prediction), but loss is computed in velocity space (v-loss). Total loss is a weighted sum of text, image, and video losses, with different weights per training stage.

Empirical Findings (F1–F8)

F1: Image Patch Size

Under equal sequence-length budget, 32×32 patches process 4× more image area per step, yet 16×16 patches consistently yield lower image generation loss. Stronger spatial compression makes generation harder to learn.

F2: Video Patch Size

Comparing p32/t4, p32/t2, p16/t4, p32/t1 tubelet configurations: stronger spatiotemporal compression leads to higher video generation loss. The final model adopts p16/t4, matching the compression ratio of common video VAEs like Wan2.2.

F3: Patch Artifacts

Linear output heads produce faint grid artifacts in smooth regions (sky, walls), exacerbated by higher classifier-free guidance (CFG). Three convolutional output heads were compared: Wan-style "upsample + conv" yields cleanest boundaries but higher training loss and blurrier images; PixelShuffle achieves the best trade-off between training loss and artifact suppression. Note: the open-source model still uses the linear output head.

F4: Pixel Space vs. VAE Space

Gradient norms are similar and training dynamics comparable; pixel-space training occasionally exhibits loss spikes. Because losses are measured in different spaces, numerical differences do not indicate faster learning in pixel space.

F5: Model Scale

The 8B model reaches the same loss in approximately one-third the training steps required by the 1.7B model.

F6: Compute Scale

Scaling from 8 to 128 GPUs reduces text cross-entropy convergence steps by ~9×, but pixel MSE convergence steps only by ~1.3×.

F7: Multimodal Context Conditioning

Feeding clean images and videos directly into the understanding expert enables image-to-video and video editing without the separate ViT and VAE encoders required by BAGEL. Adding these conditional generation tasks does not systematically degrade understanding performance.

F8: Video Understanding Interface

Two input modes are supported: dense_mode (4 FPS sampling, 4 frames per tubelet) and sparse_mode (1 FPS sampling, per-frame independent input), both with the same per-frame resolution budget (448²). Higher frame-rate dense_mode does not consistently improve results: MVBench +0.07, Video-MME −0.22, LongVideoBench −0.68, LVBench flat. Thus, 4 FPS tubelet input performs on par with 1 FPS per-frame input.

Benchmarks

Image Understanding

Evaluated with LMMS-Eval. PixelUMM achieves the best results in the table on CountBench (94.30) and PixMo-Count (73.03).

Video Understanding

Among unified models, PixelUMM leads on MVBench (70.53), Video-MME (57.33), LongVideoBench (59.61), and LVBench (40.41). Evaluation uses sparse_mode with up to 96 frames; Video-MME scores are without captions; other model scores are taken from their original papers.

Image Generation

GenEval score 0.77, improving to 0.83 with BAGEL's long-prompt rewriting. DPG-Bench total 85.74, on par with other unified models.

Video Generation

VBench total 83.24 (Quality Score 84.10, Semantic Score 79.80), close to other unified models. Imaging Quality (67.63), Temporal Style (25.84), and Overall Consistency (27.90) rank first among unified models.

Conclusion

PixelUMM unifies image and video understanding and generation by projecting image patches and video tubelets directly into a shared multimodal Transformer via a single linear layer, eliminating ViT and VAE while matching strong baselines. Experiments show that weaker spatiotemporal compression generally yields lower generation loss, and convolutional output heads can suppress patch artifacts. Future work may explore deeper MoE fusion of understanding and generation, larger patches, and longer tubelets.