How Janus‑Pro Redefines Multimodal AI with Bigger Models and New Training Strategies

Overview

DeepSeek has open‑sourced the next‑generation unified multimodal model Janus‑Pro, available in two sizes (Janus‑Pro‑1B and Janus‑Pro‑7B). The model improves both multimodal understanding and text‑to‑image generation, delivering performance comparable to or better than task‑specific models from major AI labs.

Architecture

The core architecture mirrors the original Janus design, emphasizing a decoupled approach for visual understanding and visual generation. Input data are first encoded into high‑dimensional feature sequences, which are then processed by a unified autoregressive transformer.

Decoupled Multimodal Understanding and Generation

For understanding, the SigLIP‑L encoder extracts semantic features from images. For generation, a VQ tokenizer converts images into discrete IDs. The concatenated feature sequence is fed into the large language model (LLM) for joint processing.

Optimized Training Strategy

Stage 1: Increase training steps on ImageNet while keeping LLM parameters frozen, allowing the model to learn pixel dependencies and generate coherent images.

Stage 2: Remove ImageNet and train directly on text‑to‑image data, improving training efficiency and overall performance.

Stage 3: Adjust the data ratio by reducing the proportion of text‑to‑image data, preserving strong visual generation capability while boosting multimodal understanding.

Data Expansion

Multimodal Understanding: Added roughly 90 million samples, including image caption datasets and data for tables, charts, and document understanding.

Visual Generation: Added about 72 million synthetic aesthetic samples, achieving a 1:1 real‑to‑synthetic data ratio, which speeds convergence and improves aesthetic quality.

Model Scaling

Janus‑Pro expands the base model from 1.5 B to 7 B parameters. The larger LLM backbone accelerates loss convergence for both multimodal understanding and visual generation, confirming strong scalability of the approach.

Repository Links

https://hf-mirror.com/deepseek-ai/Janus-Pro-7B
https://hf-mirror.com/deepseek-ai/Janus-Pro-1B
https://github.com/deepseek-ai/Janus

Performance Highlights

Benchmarks show Janus‑Pro surpasses previous unified models and matches or exceeds task‑specific models from Meta, Google, OpenAI, Stability AI, and others on both understanding ("Und.") and generation ("Gen.") metrics. Models that incorporate external pretrained diffusion models are marked with †.