How LongCat-Next Redefines Multimodal AI with Discrete Tokens

01 Multimodal Translation Dilemma

Most existing multimodal models translate images and audio into continuous feature vectors and then project them into a language model’s embedding space, which inevitably loses information and reduces efficiency. LongCat-Next proposes to treat all modalities as discrete tokens so that they share the same token space from the start.

02 Discretizing Vision Is Harder Than Expected

Vision is continuous and high‑dimensional, making direct tokenization challenging. The LongCat team created dNaViT , which relies on a Semantic‑Aligned Encoder (SAE) trained on large image‑text pairs to produce visual “words”. They then apply Residual Vector Quantization (RVQ) with multiple layers to compress the representation, achieving up to 28× compression while preserving semantics.

03 Audio Tokenization and Cross‑Modal Alignment

The audio branch uses a Whisper‑based encoder followed by an 8‑layer RVQ to produce discrete audio tokens. An internal language‑guidance mechanism aligns each audio segment with a corresponding text prompt, enabling both serial (text‑then‑audio) and parallel (simultaneous) generation.

04 Benchmark Results Break Conventional Biases

On visual tasks such as STEM reasoning, OCR, and document understanding, LongCat‑Next matches or surpasses specialized vision models, topping MathVista (83.1) and MathVision (64.7). For text‑to‑image generation, it outperforms dedicated models on GenEval, DPG‑Bench, and LongText.

Audio benchmarks show MMAU 76.40, ASR word‑error rate 1.47 % on AISHELL‑1, and TTS SeedTTS Chinese score 1.90, rivaling top‑tier models like Gemini 3.1 Flash‑Lite and Qwen‑3 Omni—all achieved within a single unified backbone.

05 Towards a Platonic Representation Hypothesis

t‑SNE visualizations reveal that LongCat‑Next’s visual and textual tokens intermix in the same embedding space, unlike non‑native models where they remain separated. The team calls this the “Platonic Representation Hypothesis”, suggesting that different modalities are merely different projections of a shared reality.

Conclusion

Unifying all modalities as discrete tokens brings three major benefits: (1) Simplified architecture and engineering—training and deployment follow the mature language‑model pipeline; (2) Capability sharing—understanding and generation become two sides of the same token‑prediction problem; (3) Data expansion—any image, text, audio, or video can be converted into a uniform token stream, unlocking larger self‑supervised datasets.

Current limitations include compute and data scale, leaving open research directions such as longer cross‑modal context, multi‑turn multimodal dialogue, and finer‑grained interactive generation.

GitHub: https://github.com/meituan-longcat/LongCat-Next
HuggingFace: https://huggingface.co/meituan-longcat/LongCat-Next
demo: https://longcat.chat/longcat-next