Skip to content
LongCat-2.5: 1.6T Native Multimodal Model for Visual Coding & Long-Horizon Agents

LongCat-2.5: 1.6T Native Multimodal Model for Visual Coding & Long-Horizon Agents

LongCat-2.5 is a 1.6-trillion-parameter native multimodal mixture-of-experts model that activates 48B parameters per token, combining multi-level discrete visual tokens with large-scale long-horizon reinforcement learning to excel at visual coding, complex repository development, and creative 3D tasks, while achieving 86% higher training throughput on domestic AI accelerators and 43% faster inference via W4A4 quantization.

Meituan Technology Team
Meituan Technology Team
Meituan Technology Team

Model Overview

LongCat-2.5 is a native multimodal mixture-of-experts (MoE) model designed for long-horizon agent tasks. It has 1.6 trillion total parameters and activates approximately 48 billion parameters per token. Building on LongCat-2.0, it strengthens native visual understanding and agent programming capabilities.

Native Visual Representation

Discrete Native-Resolution Vision Transformer

Visual signals are represented as multi-level vocabulary tokens, enabling the backbone to process vision and text uniformly. Following the LongCat-Next method [1], a Semantic-and-Aligned Encoder (SAE) produces native-resolution features, which are then discretized via Residual Vector Quantization (RVQ) to address the information bottleneck of visual discretization.

Figure 1: Native-resolution visual discretization: encoding images of different resolutions into discrete token sequences and reconstructing via a visual decoder.
Figure 1: Native-resolution visual discretization: encoding images of different resolutions into discrete token sequences and reconstructing via a visual decoder.

Offline Visual Token Preprocessing

A fixed tokenizer allows offline precomputation and caching of visual token IDs. The backbone then directly processes pre-tokenized sequences, simplifying long-sequence packing and splitting, and improving load balancing in distributed training.

Native Visual Embeddings

Visual codebook embeddings are jointly learned with the backbone. On a smaller MoE model with ~3B average activated parameters, t-SNE visualization shows interleaved distribution of visual and text embeddings, indicating fusion in a shared semantic space.

Figure 2: t-SNE comparison of visual and text embeddings. Right panel shows validation on a ~3B activated-parameter MoE model.
Figure 2: t-SNE comparison of visual and text embeddings. Right panel shows validation on a ~3B activated-parameter MoE model.

Agentic RL Training Pipeline

Large-scale reinforcement learning (RL) scaling is central to improving agent intelligence. LongCat-2.5 builds a comprehensive agentic RL training pipeline covering data, environment, algorithm, and Harness.

Figure 3: Agentic RL training pipeline for long-horizon tasks, covering data and environment, asynchronous training, reward design, and mixed training.
Figure 3: Agentic RL training pipeline for long-horizon tasks, covering data and environment, asynchronous training, reward design, and mixed training.

Large-Scale Data Synthesis and Environment Construction

An automated pipeline generates training tasks, interactive environments, and verifiable test cases from GitHub repositories and internal code tasks. For selected PRs and linked issues, an agent automatically labels, synthesizes environments, and performs fail-to-pass (F2P) and pass-to-pass (P2P) test verification. Real-time rollout trajectories and pass rates assist in secondary labeling and selection; false-negative/false-positive analysis continuously refines F2P test cases to improve functional coverage and verification signal accuracy.

Asynchronous RL and Stability Optimization

A partial rollout asynchronous strategy mitigates long-tail inefficiency: each RL step maintains an oversampling rate, interrupts when a batch is formed, saves incomplete trajectories, and resumes in the next sampling phase. Oversampling rate adapts to task pass rate; zero-gradient groups (all-correct or all-wrong) are filtered. To handle staleness from tokens originating from different historical checkpoints, the logprob difference between sampling and training is clipped, and per-token gradient contributions are adjusted based on cumulative historical difference, limiting gradient bias. Infrastructure-level train-inference consistency optimizations further stabilize training.

Reward Hacking Mitigation and Output Behavior Optimization

Speculative behavior is constrained by disconnecting network, clearing git history and file caches in the task environment, and introducing an LLM judge to supervise trajectories. Repetitive thinking and tool calls receive extra penalties; overthinking and rapid length growth are suppressed by redistributing reward among positive samples.

Mixed Training and Generalization

The model autonomously learns and adapts to similar tasks and Harnesses during RL. A brief cold-start phase covers only a few scaffold adaptation cases; cross-task and cross-scaffold mixed training in the RL phase significantly improves generalization and boosts performance across tasks.

Expert Group Mechanism and Online Distillation

Following LongCat-2.0, domain-specific expert groups optimize agent capability, reasoning, interaction experience, and safety without sacrificing helpfulness. In Multi-Objective Policy Distillation (MOPD), the student model samples online on domain tasks, and domain experts train the student via full-vocabulary KL divergence [2] on the sampling results.

To limit distribution shift, teacher trajectories from SFT or RL are split into per-turn prefixes as student sampling context. LongCat-2.5 further splits each teacher turn into fixed-length per-cut prefixes and caps student self-sampling length to that value, addressing distribution shift within single long turns.

Figure 4: Per-cut prefixes used for student online sampling.
Figure 4: Per-cut prefixes used for student online sampling.

Prefixes from the same teacher trajectory are grouped and sent to a single inference engine to share prefill computation and KV cache. Training constructs a tree with teacher trajectory as trunk and student samples as branches, combined with Tree Attention [3], enabling a single forward-backward pass for all samples in the group, avoiding redundant computation and storage on the teacher trajectory.

Long-Context Training Infrastructure on Domestic AI Accelerators

Both mid-training and supervised fine-tuning use long-sequence training. With domestic accelerator memory under 80 GB per GPU, the system optimizes memory and designs parallelism around super-node topology.

Super-node topology and parallelism configuration.
Super-node topology and parallelism configuration.

FSDP and expert parallelism (EP) degrees match the number of GPUs per super-node; context parallelism (CP) degree does not exceed that number, keeping parameter AllGather, expert token AllToAll, and KV AllGather within the high-bandwidth intra-super-node domain.

Hybrid vocabulary parallelism shards the base embedding, LM head, and N-gram embedding layers across the combined DP and CP domain, further reducing memory.

Dynamic expert replication based on MoonEP triggers only when EP load imbalance exceeds a threshold, avoiding extra communication overhead and preventing receiver OOM from routed tokens or sender OOM from communication ops.

Early image token generation removes the vision encoder from the training graph, simplifying parallelism. Known image token counts allow pre-estimation of modality load ratios during data packing and shuffling, balancing modality distribution across DP ranks.

Dual-shard FSDP: model parameters sharded within super-node FSDP domain; FP32 master weights and optimizer states further sharded across FSDP domains via HSDP replicas, maintaining parameter communication efficiency while reducing resident memory.

Parameters, gradients, and optimizer states organized as contiguous flattened buffers, supporting non-uniform sharding and Muon optimizer sharding. Direct weight gradient writes and temporary buffer reuse reduce intermediate copies and peak memory.

Zero-copy AllGather/ReduceScatter eliminates intermediate copies from default communication algorithms. EP packs tokens with routing probabilities and uses AIV-based communication to alleviate AICPU task launch bottlenecks. Parameter prefetching overlaps AllGather with compute while controlling CP resource contention; ScMoE overlaps EP communication with dense branch compute.

Batched gradient operations, merged metric reductions, CPU pinning, and NUMA-aware scheduling reduce host-side overhead.

Combined, this FSDP configuration achieves 86% higher single-GPU throughput than a DDP baseline on long-sequence SFT tasks .

Figure 5: Long-context training on domestic AI accelerators.
Figure 5: Long-context training on domestic AI accelerators.

Train-Inference Consistency

Based on IsoExec [4], GLM-5 [5], and XoRL [6], a production-grade consistency scheme enables independent parallel configurations for training, prefill, and decode while maintaining bitwise per-token logprob consistency with near-zero performance loss. Both ends use efficient batch-invariant operators and align framework compute logic, adding backward support. The scheme is enabled by default in experiments and production, providing a unified numerical regression baseline for operator and framework iterations. TP and DCP consistency support are extended beyond community solutions.

Train-inference consistency configurations.
Train-inference consistency configurations.

Tensor Parallelism (TP)

Replaces segmented reduction with full K-dimension GEMM, near-zero performance loss. Training retains weight sharding; weight All-Gather and input AlltoAll enable independent full K-dim GEMM without Reduce-Scatter sum. Prefill uses same scheme; Decode changes o_proj/fc2 to column-parallel, using input All-Gather, full K-dim GEMM, and output AlltoAll, avoiding per-step full weight aggregation and cross-GPU numerical reduction.

Dual Chunk Parallelism (DCP)

Replicates Decode's accumulation path in training forward and prefill. Matching sparse attention operators are inserted in training forward and prefill to reproduce the KV chunking and reduction order of a given DCP configuration at Decode, aligning compute precision across all three phases.

Long-Context Multi-Turn Rollout Optimization

For controllable-load long-context multi-turn rollout, prefill-decode (PD) separation assigns suitable parallel strategies to each, with co-optimization across cache storage, cross-layer offloading, and request scheduling.

Figure 6: Prefill/Decode configurations for tested long-context loads; 3:2 P/D instance ratio optimal.
Figure 6: Prefill/Decode configurations for tested long-context loads; 3:2 P/D instance ratio optimal.

Expand effective cache capacity: Prefill uses QCP (token-level compute allocation) and KVP (page-based KV storage) to avoid KV redundancy from MLA TP splitting. HiCache releases host-memory copies after KV migration to GPU memory, reducing duplicate two-level cache occupancy. Mooncake-based weight management and actor offload free more memory for HiCache.

Hide transfer overhead via cross-layer prefetch: Leveraging LSA [7] where odd layers reuse top-k indices from previous even layers, odd-layer full KV is offloaded to host memory, keeping only a small working cache in GPU memory. After even layer produces indices, the next odd layer's required KV is prefetched early and overlapped with current layer compute, nearly doubling decode KV cache capacity without increasing TPOT.

Prevent recomputation avalanches via capacity-aware scheduling: Router controls request admission based on L1+L2 cache capacity and total request tokens, prioritizing high-completion groups to quickly form training batches and release cache, avoiding repeated eviction of each other's prefixes when multiple long trajectories interleave, which would trigger massive recomputation.

W4A4 Efficient Inference for 1.6T MoE

Combining model structure and domestic AI accelerator hardware characteristics, a collaborative optimization across quantization precision, low-level operators, and inference pipeline yields a W4A4 solution.

W4A4 inference optimization overview.
W4A4 inference optimization overview.

Precision: Based on FlatQuant's learnable equivalent transform, further optimized for MoE models, maintaining performance on par with pre-quantization model across evaluated downstream tasks.

Operator and data path: Expert-shared input transform completes quantization before token replication and distribution, so a single INT4 representation flows through expert communication and compute. Transform, scale computation, quantization, and packing are fused into a dedicated operator, combined with weight layout and transpose access optimization, reducing data movement and conversion overhead.

System-level: Pipeline overlapping of compute and communication optimized for prefill and decode based on data dependencies and hardware constraints.

In online stress tests, relative to a W8A8 INT8 baseline , prefill throughput increases 43.3% , decode throughput increases 42.77% , TTFT decreases 15.91% , and TPOT decreases 19.72% . At the operator level, GMM achieves an average 1.73× speedup over W8A8.

W4A4 inference benchmark results.
W4A4 inference benchmark results.

Demonstrations

Real end-to-end generations showcase software development and professional office scenarios: real-time fluid simulation and visual effects generation, 3D digital exhibition (e.g., Sanxingdui bronze sacred tree), city sandbox with isometric 3D viewport and low-poly lighting, business analysis integrating multi-channel data for profit calculation and actionable recommendations, feedback report generation with interactive HTML, traditional Chinese color aesthetics dynamic webpage, miniature scene (carousel) with multi-material rendering, and reusable event invitation tool generating signup pages from venue photos and natural language.

References

[1] LongCat-Next method: https://arxiv.org/abs/2603.27538

[2] Full-vocabulary KL divergence: https://huggingface.co/deepseek-ai/DeepSeek-V4.1-Flash/blob/main/DeepSeek_V41_Tech_Report.pdf

[3] Tree Attention: https://arxiv.org/pdf/1909.07606

[4] IsoExec: https://vllm.ai/blog/2026-08-21-isoexec

[5] GLM-5: https://github.com/THUDM/slime/pull/2262

[6] XoRL: https://kiddyboots216.github.io/mismatch/

[7] LSA: https://arxiv.org/abs/2608.01662

All results obtained with enable_thinking=on, temperature=1.0, top_p=0.95. Scores marked * from public technical reports, blogs, or official leaderboards; unmarked scores are self-evaluated for reference. Details at https://longcat.ai/blog/longcat-2.5.

Original Source

Signed-in readers can open the original source through BestHub's protected redirect.

Sign in to view source
Republication Notice

This article has been distilled and summarized from source material, then republished for learning and reference. If you believe it infringes your rights, please contactadmin@besthub.devand we will review it promptly.

Mixture of ExpertsReinforcement LearningMultimodal ModelVisual CodingLong-Horizon AgentsLongCat-2.5Train-Inference ConsistencyW4A4 Quantization
Reader feedback

How this landed with the community

Sign in to like

Rate this article

Was this worth your time?

Sign in to rate
Discussion

0 Comments

Thoughtful readers leave field notes, pushback, and hard-won operational detail here.