SenseNova U1: Open‑Source SOTA Multimodal Model Unifies Vision and Language

Architecture and Innovations

SenseNova U1 replaces the traditional multimodal pipeline that stacks a visual encoder (VE) and a variational auto‑encoder (VAE) with a native NEO‑unify architecture. Pixels and text are modeled in a shared representation space, eliminating intermediate translation steps.

Near‑lossless visual interface : inputs and outputs remain at pixel‑level fidelity without compression trade‑offs.

Mixture‑of‑Transformer (MoT) : a single backbone jointly handles visual understanding and generation, reducing modality conflicts and computational overhead.

Unified learning objective : text is trained with auto‑regressive cross‑entropy, vision with pixel‑flow matching; both objectives are optimized within the same training loop.

Model Variants and Training Pipeline

Two specifications are open‑sourced: SenseNova U1‑8B‑MoT: dense 8‑billion‑parameter backbone. SenseNova U1‑A3B‑MoT: MoE‑based variant (download not yet available).

The training follows a staged process:

Understanding warm‑up.

Generation pre‑training.

Unified mid‑training.

Supervised fine‑tuning.

One round of text‑to‑image reinforcement learning (RL) on the base model.

Despite the modest parameter count, the model achieves a “hard” benchmark score sheet.

Benchmark Performance

SenseNova U1 attains state‑of‑the‑art results on a wide range of multimodal understanding, generation, and reasoning tests. On OneIG (EN, ZH), LongText (EN, ZH), CVTG, BizGenEval (Easy, Hard), and IGenBench, it shows low generation latency while delivering higher average performance than previous open‑source models of comparable size.

On information‑graphics benchmarks (BizGenEval, IGenBench) the model maintains low latency and superior average scores.

Unified Capabilities

Unlike the traditional two‑step pipeline (text → image), SenseNova U1 performs simultaneous understanding and generation. Example prompts demonstrate the model’s abilities:

Reasoning‑driven generation: the prompt “a small piece of dry wood and a dense iron block in a transparent water tank” yields a physically consistent image.

World‑knowledge infographics: generating detailed diagrams that embed factual information.

Image editing: swapping water with high‑concentration salt water and observing floating eggs.

Interleaved text‑image story generation across multiple pages.

Interactive generation respecting physical constraints, e.g., moving a green‑bordered rectangle horizontally to align under a red‑starred blue circle.

Known Limitations

The current version supports up to 32K visual tokens; extremely long or complex scenes strain the model. Human detail generation degrades when the person occupies a small region or interacts intricately with other objects. Text rendering can suffer from misspellings, deformation, and sensitivity to prompt phrasing, especially at high density. The mixed generation mode is still experimental and lags behind dedicated text‑to‑image pipelines; RL fine‑tuning has not yet been specialized for editing, reasoning, or interleaved tasks.

Future Directions

The architecture leaves room for extensions to visual‑language‑action (VLA) and world‑model learning. Larger‑scale versions are planned, with targeted data and training enhancements to address the listed shortcomings.

Resources

Repository and model weights:

GitHub: https://github.com/OpenSenseNova/SenseNova-U1

HuggingFace collection: https://huggingface.co/collections/sensenova/sensenova-u1

ModelScope collection: https://www.modelscope.cn/collections/SenseNova/SenseNova-U1

NEO‑unify blog post: https://huggingface.co/blog/sensenova/neo-unify