Can Uni‑X Eliminate Multimodal Gradient Conflict with a Pure Autoregressive Design?

Uni‑X, a paper accepted at ICLR 2026, tackles the gradient‑conflict problem that arises in unified multimodal models (UMMs) when a single autoregressive Transformer processes both text and visual inputs.

Motivation. Converting visual inputs into discrete tokens via vector quantization and concatenating them with text is the dominant AR‑UMM approach. Empirical analysis shows that a standard fully‑shared Transformer encounters severe gradient conflict , especially in the shallow and deep layers, when handling such multimodal sequences.

To quantify the phenomenon, the authors compute the cosine similarity between gradients obtained from pure‑text data and from image‑text data, subtract the baseline similarity under mixed‑modality distribution, and observe extreme conflict in the early and final layers, with a partial relief in the middle layers.

Source of Conflict. From an information‑theoretic perspective, visual token sequences have a much higher conditional entropy than natural language (measured via N‑gram entropy). This high‑entropy visual stream forces the shallow layers, which are responsible for low‑level feature extraction, to reconcile fundamentally different statistical properties, leading to strong gradient tug‑of‑war. In contrast, middle layers produce more abstract, semantic representations where modalities align more naturally, reducing conflict.

Uni‑X Architecture. The proposed solution is a “two‑end‑separated, middle‑shared” X‑shaped architecture. The bottom (encoder‑like) and top (decoder‑like) layers are split into parallel, modality‑specific branches, ensuring that text and visual streams are processed independently during early feature extraction and final token projection. The intermediate layers remain shared, focusing on high‑dimensional cross‑modal fusion and reasoning.

This design eliminates the need for external visual encoders (e.g., CLIP) and aligns with classic encoder/decoder concepts, although the authors did not conduct dedicated ablation experiments for that analogy.

Theoretical Efficiency Gain. Because visual and textual tokens are strictly isolated in the separated layers, the self‑attention complexity for a sequence of length \(L_{v}+L_{t}\) drops from \(O((L_{v}+L_{t})^{2})\) to \(O(L_{v}^{2}+L_{t}^{2})\), providing a higher theoretical throughput ceiling for the same parameter budget.

Experimental Results. Under identical training budgets, the 3B‑parameter Uni‑X demonstrates strong scaling and competitiveness:

Image Generation & Understanding: Without any extra semantic encoder, Uni‑X scores 82 on the GenEval benchmark, matching or surpassing several 7B‑scale autoregressive UMMs.

Zero‑Shot Image Editing (ImgEdit): Fine‑tuned on only ~90k image‑editing examples, Uni‑X achieves performance comparable to the larger Bagel model that uses more data and parameters.

Future Work. The authors acknowledge that not using an external visual feature extractor limits the ultimate multimodal understanding ceiling. They plan to explore removing the VQ‑VAE tokenizer entirely, allowing the X‑branch to perform direct pixel‑to‑pixel tokenization and detokenization for a truly end‑to‑end native multimodal model.