Janus-Pro Unveiled: A Unified Architecture for Multimodal Understanding and Generation

Overview

Recent advances in unified multimodal understanding and generation models have shown impressive gains, but most rely on a single visual encoder for both tasks, leading to sub‑optimal performance for understanding. Janus‑Pro addresses this by decoupling visual encoding, alleviating conflicts between the two tasks and delivering strong results across the board.

Introduction

Janus‑Pro is an upgraded version of the earlier Janus system. It incorporates (1) optimized training strategies, (2) expanded training data, and (3) larger model scales. These improvements markedly enhance multimodal understanding and instruction‑following text‑to‑image generation while also stabilizing generation quality. The code and models are publicly released.

Architecture

The architecture (see Figure 3) follows the same overall design as Janus but isolates visual encoding for understanding and generation. Raw inputs are first transformed into features by independent encoders. For understanding, a SigLIP [53] encoder extracts high‑dimensional semantic features, flattens the 2‑D grid to a 1‑D sequence, and maps them to the LLM input space via a comprehension adapter. For generation, a VQ tokenizer [38] converts images to discrete IDs; a generation adapter maps the codebook embeddings to the LLM space. Both feature streams are concatenated into a multimodal sequence processed by a unified autoregressive transformer, which also includes a randomly initialized prediction head for visual generation.

Optimized Training Strategy

The original Janus used a three‑stage training pipeline. Stage 1 trained adapters and the image head; Stage 2 performed unified pre‑training, updating all components except the comprehension encoder; Stage 3 applied supervised fine‑tuning, unlocking the comprehension encoder. In Stage 2, text‑to‑image training was split into two parts following PixArt [4]: 66.67 % of steps used ImageNet class‑name prompts, the rest used regular text‑to‑image data, which proved inefficient.

Janus‑Pro makes two key changes:

Longer Stage 1: more steps on ImageNet allow the model to learn pixel dependencies even with a frozen LLM, producing reasonable images from class names.

Focused Stage 2: the ImageNet portion is removed; the stage now trains directly on ordinary text‑to‑image data, improving efficiency and overall performance.

Additionally, the data‑mix ratios in Stage 3 are altered from 7:3:10 (multimodal:pure‑text:text‑to‑image) to 5:1:4, slightly reducing text‑to‑image data while preserving strong generation capability and boosting understanding.

Data Expansion

Janus‑Pro enlarges both multimodal understanding and visual‑generation datasets.

Understanding: Stage II pre‑training adds ~90 M samples drawn from DeepSeek‑VL2 [49], including image‑caption data (e.g., YFCC [31]) and tabular/document data (e.g., Doc‑matix [20]). Stage III fine‑tuning further incorporates MEME understanding, Chinese dialogue, and other dialogue‑enhancing corpora.

Generation: ~72 M synthetic aesthetic samples are introduced, achieving a 1:1 real‑to‑synthetic ratio during unified pre‑training. Prompts are publicly available (e.g., from [43]). Experiments show faster convergence and markedly higher aesthetic quality on synthetic data.

Model Scaling

While Janus proved the effectiveness of a 1.5 B LLM with decoupled visual encoding, Janus‑Pro scales the backbone to 7 B. Table 1 (not reproduced) lists hyper‑parameters for both 1.5 B and 7 B variants. Larger LLMs converge faster on both understanding and generation losses, confirming strong scalability.

Experiments and Results

Implementation Details

Experiments use DeepSeek‑LLM (1.5 B and 7 B) [3] with a maximum sequence length of 4096. SigLIP‑Large‑Patch16‑384 [53] serves as the visual encoder for understanding; the generation encoder employs a codebook of size 16 384 with 16× down‑sampling. Both adapters are two‑layer MLPs. Stage II employs early stopping at 270 K steps. Images are resized to 384 × 384; for understanding data, the longer side is scaled and the shorter side padded with RGB (127,127,127). Generation data are center‑cropped to 384. Training uses sequence packing and mixed‑data batching. Janus‑Pro is trained and evaluated with HAI‑LLM [15], a lightweight PyTorch‑based distributed framework, across 16–32 nodes (each with eight Nvidia A100 40 GB GPUs) for roughly 9–14 days.

Evaluation Setup

Multimodal understanding is assessed on standard vision‑language benchmarks such as GQA. Visual generation is evaluated with GenEval [14] and DPG‑Bench [16]; GenEval provides instance‑level analysis of text‑to‑image capability, while DPG‑Bench contains 1 065 dense prompts to test complex semantic alignment.

Comparison with State‑of‑the‑Art

On understanding benchmarks (Table 3), Janus‑Pro achieves the best overall scores, outperforming larger models like TokenFlow‑XL (13 B) on all tests except GQA. The advantage stems from the decoupled visual encoder, which reduces task conflict.

For generation (Table 4), Janus‑Pro‑7B reaches 80 % overall accuracy on GenEval, surpassing Transfusion [55] (63 %), SD3‑Medium (74 %), and DALL‑E 3 (67 %). On DPG‑Bench (Table 5), Janus‑Pro scores 84.19, the highest among all compared methods, demonstrating superior dense instruction following.

Qualitative Results

Figure 4 shows Janus‑Pro’s strong comprehension across diverse contexts and high‑fidelity 384 × 384 text‑to‑image outputs. Even with limited resolution, the model renders detailed scenes and captures nuanced semantics in imaginative prompts.

Conclusion

The paper analyzes Janus‑Pro’s gains from three perspectives: training strategy, data expansion, and model scaling. Limitations include the fixed 384 × 384 input resolution, which hampers fine‑grained tasks like OCR, and occasional lack of detail in small facial regions of generated images. Increasing resolution is a promising direction for future work.