Agentic RL Infra Survey: 11 Chinese Labs' Approaches to Long-Horizon Credit Assignment

Background: From RLHF to Agentic RL

The post-training paradigm has rapidly evolved from RLHF (single-turn dialogue with scalar preference rewards) to RLVR (verifiable rewards like math/code tests) to Agentic RL, where agents perform multi-turn tool use, file I/O, code execution, web retrieval, and long-horizon tasks in real or simulated environments. This shift exploded in 2026. The 11 surveyed teams—Xiaomi MiMo, DeepSeek, Moonshot Kimi, Zhipu GLM/Z.ai, Alibaba Qwen, MiniMax, StepFun, Meituan LongCat, Ant Ling/Ring, ByteDance Seed, Tencent Hunyuan—all target agentic coding (SWE), terminal operation, deep search, GUI/multimodal operation, and long-horizon assistant workflows. Xiaomi MiMo-V2.6 even proposes "You Only RL Once" training code/general/visual/cyber domains in a single mixed run. DeepSeek-V4.1 asserts that engineering data and environment pipelines now yields higher marginal returns than algorithmic innovation in post-training.

Four Core Challenges for Agentic RL Infra

Trajectories grow long : single trajectories span hundreds to thousands of tool calls, accumulating millions of tokens. Kimi K3 RL directly targets 1M context; Qwen3-Coder-Next SWE RL raises average interaction turns from 50 to 130.

Environments grow heterogeneous : rollout interleaves code sandboxes, browsers, MCP tools, terminal sessions—each with startup latency, failure rates, and non-determinism.

Rewards grow sparse : a hundreds-step trajectory often yields only a final binary pass/fail; assigning credit to intermediate steps becomes the central difficulty.

Heavy-tailed duration : ~5% of samples consume ~80% of generation compute; synchronous waiting for the slowest trajectory leaves expensive training GPUs idle.

The core contradiction: how to keep training compute fully utilized and gradient signals trustworthy in an environment full of long tails and noise ; all research explores the throughput vs. off-policy trade-off.

System Overview: Four Interlocking Planes

Unlike static pre-training/SFT pipelines, Agentic RL forms a continuous closed loop where training data is produced online by the current policy interacting with external environments. A complete system decomposes into:

Control Plane : global scheduling, resource allocation, version management, failure recovery, observability.

Data Plane : trajectory, logprob, reward flow between Rollout, Queue, Verifier, and Trainer.

State Plane : KV cache, environment state, tool sessions, partial rollout state persistence and recovery.

Weight Plane : new model weights synchronization, conversion, and version switching from Trainer to Rollout Engine.

Key Problems and Mainstream Solutions

3.1 Architecture: Colocated (共卡) vs. Disaggregated (分离)

Rollout is memory-bound, high-concurrency, heavy-tailed inference; training is compute-bound, synchronous. Colocated shares GPUs time-sliced—saves machines and weight transfer but forces KV cache eviction during training, couples timing, limits throughput ceiling. Disaggregated uses separate GPU pools—enables independent scaling and optimal parallelism per workload, but incurs cross-cluster weight sync latency and resource partitioning complexity. In Agentic RL, single-trajectory length and environment wall-clock time amplify GPU idleness, making architecture choice critical.

Mainstream camps:

Colocated : Kimi K3 (offloads weights/optimizer to NVMe, yields CPU DRAM to external KV cache pool), Tencent Hunyun Hy3 (Megatron training + vLLM rollout, full offload to CPU during rollout), DeepSeek-V4.1 (colocated + external DSec sandbox platform), Ant ASystem (Hybrid Runtime, zero-redundancy P2P weight exchange via AState, 10s for trillion params), Xiaomi MiMo-V2.6 (colocate_async: interrupt in-flight sequences when batch ready, re-prefill on resume).

Disaggregated : GLM-5 (decoupled train/inference GPUs, Multi-Task Rollout Orchestrator, 1k+ concurrent rollouts, micro-service scaffolds), MiniMax Forge (Middleware: Gateway Server + Data Pool, fully decoupled pipelines), LongCat DORA (RolloutManager, SampleQueue, Trainer on different nodes, 32k concurrent envs, ~400 physical machines), Qwen (fully disaggregated, 3-5x end-to-end speedup via system-algorithm co-design).

Elastic colocated hybrid : LongCat DORA (dedicated generation group + elastic role group that switches per step).

Consensus : regardless of architecture, environment/sandbox execution moves to CPU-side elastic scaling (DeepSeek DSec, Ling's thousands of ASandbox replicas, LongCat CPU-idleness-aware RPC) because environment latency is dominated by external systems and should not occupy GPUs.

Framework stack converges on Megatron-LM training + vLLM/SGLang rollout + Ray orchestration , with each team building custom orchestration/decoupling layers: GLM slime, MiniMax Forge, LongCat DORA, Ant ASystem, StepFun Steptron/Session-Router, Qwen MegaFlow.

3.2 Sync vs. Async, Partial Rollout & Long-Tail Mitigation

Synchronous RL waits for all trajectories in a batch; Agentic trajectories exhibit extreme length variance (MiMo: 90x token count difference across 25 data sources, 66x active duration; StepFun: 5% samples consume 80% generation cost). This creates massive GPU bubbles. Async (fully async, partial/streaming rollout) becomes inevitable but introduces off-policy effects.

Three schools:

Colocated async (partial rollout) : generate/train alternate on same GPUs; pause all in-flight sequences when batch ready, train, swap weights, resume. Trajectories span multiple policy versions—"async" means cross-version generation, not temporal overlap. Kimi K3 (λ proportion pause/resume), MiMo (GRPO + async partial rollout, staleness=4, re-prefill cost), DeepSeek-V4.1 (sample-level scheduling, pause at any token boundary, resume from persisted state without re-prefill), Ring-1T C3PO++ (global token budget Φ per round).

Disaggregated fully streaming async : inference pool continuously generates, completed trajectories enqueued; training pool consumes batches, inference continues on old weights, new weights pushed every K steps. GLM-5, LongCat DORA (2-4x faster than sync), StepFun (10x efficiency gain after moving to FullyAsync).

Disaggregated async + windowed sampling : MiniMax Windowed FIFO (W=0.3N) limits max off-policy lag while reducing head-of-line blocking.

Shared issue: length bias —short trajectories finish first, dominate early batches, bias model toward short outputs. DeepSeek mitigates by per-dataset concurrency limits and discarding early-return short samples.

3.3 Staleness / Off-Policy Control

Async trajectories may span multiple model versions—Kimi K3 calls this "extreme off-policy". Behavior policy (generation) vs. target policy (training) version gap (staleness) must be bounded to avoid instability. Three approaches:

Version cap + discard : GLM-5 tracks all policy versions per trajectory; discards if oldest version > τ. Ring-1T C3PO++ uses retention period σ (number of cuts); discards rollouts with σ > threshold.

Staleness manager admission control : Ling 2.6 tags each rollout segment with generation-time policy version; limits in-flight pool size: pool_size ≤ max_staleness × consumer_batch_size, bounding effective staleness to a few optimization steps. Requires sub-second weight broadcast.

Algorithmic masking/truncation : no explicit version tracking; mask or truncate importance sampling (IS) ratios for tokens with excessive deviation. Includes GLM DIS, MiMo four-boundary masking, StepFun MIS-PO, LongCat HisPO (detailed in §3.4).

3.4 Training-Inference Mismatch

Generation uses inference engines (vLLM/SGLang), optimization uses training engines (Megatron). Even with bitwise identical parameters, kernel implementations, numeric precision, and reduction order differ, causing logprob divergence. MoE expert routing is especially fragile: Qwen measured ~10% expert activation change per RL update on Qwen3-30B-A3B. This logprob difference accumulates over long sequences, gets amplified by clipping, and becomes the primary source of gradient variance and training collapse. Ring-1T proves a "compound probability divergence theorem": per-step divergence lower-bounded by geometric growth, so divergence never self-corrects.

Four solution categories (often stacked):

Algorithmic correction (masking/truncation) : IcePop (Ring-1T), KPop (Ling), DIS (GLM), MIS-PO (StepFun), HisPO (LongCat)—gate on ρ_ti = π_train(a_t|s_t) / π_infer(a_t|s_t) (same version, two engines) to decide token inclusion, then clip full IS ratio ρ_ti * ρ_ti^stale for step-size control.

Sequence-level IS bypass : Qwen GSPO uses length-normalized sequence-level IS ratio ρ_seq = exp(1/L Σ log ρ_ti), making optimization unit match reward unit (sequence), eliminating token-level routing sensitivity and removing need for routing replay.

System-level alignment (eliminate bias at source) : DeepSeek bitwise alignment (batch-invariant attention, DeepGEMM, deterministic reduction, native FP4 rollout); MiMo/Kimi MXFP4 QAT shared quantization; MiniMax/Ling FP32 LM Head (2-point reward gain); GLM deterministic torch.topk replacing non-deterministic CUDA top-k.

Forced routing consistency (R3) : MiMo R3 records rollout expert indices, replays in training; DeepSeek-V4.1 concatenates per-segment routing for partial rollout; StepFun Routing Confidence metric quantifies need for replay.

Highest ROI single fix : FP32 LM Head (MiniMax, Ling independently discovered, ~2 reward points). Algorithmic and system approaches are not mutually exclusive—MiMo stacks QDQ alignment, R3 replay, and four-boundary masking.

3.5 Policy Optimization Algorithm Selection

GRPO (group-relative advantage, no critic) is de facto baseline but exhibits four pathologies under long sequences, MoE, async, agentic multi-turn:

Granularity mismatch : sequence-level reward, token-level IS weights—unequal weights unrelated to token contribution accumulate irreversibly (Qwen GSPO paper).

Clip systematically favors dropping low-prob tokens : MiniMax found tokens like "wait" permanently clipped; DAPO proved upper clip bound 1+ε allows high-prob tokens unbounded increase but low-prob tokens capped at (1+ε)p_old, making clipping candidate set structurally low-prob.

Group sampling incompatible with async/agentic : group waits for slowest (latency-driven off-policy); many online envs give only one trajectory per prompt (GLM SAO).

No critic → coarse credit assignment : hundreds of tokens share single sequence advantage; SAO shows vanilla GRPO collapses ~160 steps, vanilla VAPO ~90 steps, but with trained value model stable to ~1000 steps.

Four key design variables : IS ratio level (token vs. sequence), clip type (saturate/min, mask/zero, truncate coefficient), KL usage, critic presence.

Representative algorithms:

DAPO (ByteDance Seed + Tsinghua AIR) : Clip-Higher (decouple ε_high/ε_low), Dynamic Sampling (filter all-correct/all-wrong prompts, refill batch), Token-Level Loss (global token-count normalization, fixes length bias), Overlong Reward Shaping (filtering + soft penalty). Ablation: Naive GRPO 30 → +Overlong Filtering 36 → +Clip-Higher 38 → +Soft Overlong 41 → +Token-Level 42 → +Dynamic Sampling 50 (AIME24 avg@32).

CISPO (MiniMax) : clips IS weight ρ_ti instead of token gradient, preserving gradient flow for all tokens; uses stop-gradient on ρ_ti; token-level normalization; asymmetric clip [0, 10] in M2.

GSPO (Qwen) : sequence-level IS, clip range 3e-4/4e-4 (vs GRPO 0.2/0.27); clips 15% tokens vs GRPO 0.13% yet higher training efficiency—evidence GRPO token gradients are noisy.

SAO (GLM/Tsinghua) : group size=1, uses critic; DIS objective (denominator = rollout logprob, double-sided mask), Skip-Observation GAE (action tokens directly connect, skipping environment observations), Frozen Attention critic, Length-adaptive GAE.

MIS-PO (StepFun) : binary Metropolis-style acceptance filter at token level ( [0.5, 2]) and trajectory level (geometric mean [0.996, 1.001]); accepted samples treated as on-policy (weight=1); retains critic with Truncation-Aware Value Bootstrapping ( V(s_T) for truncated trajectories).

CompactionRL (Tsinghua) : for context compaction (segments share task reward), token-level loss normalization + Cross-Trajectory GAE (discount by future segment token counts).

DIS (GLM) : standalone component; swaps IS denominator to rollout logprob, no historical version tracking needed. Ablation: GRPO 84.2/54.8 → +DIS 93.5/70.8 (AIME2025/BeyondAIME).

3.6 Sparse Reward & Credit Assignment

Agentic trajectories: hundreds of steps, single binary pass/fail. Two problems: (1) no discrimination among passing solutions; (2) cannot identify which step/turn succeeded/failed. MiMo makes grader compute a third scaling dimension (12.7% of Pro cost).

Four schools:

Outcome + reward shaping : MiniMax composite reward R = R_task + λ1 R_process + λ2 R_time (process penalizes language mixing/tool format errors; time uses monotonic wall-clock penalty, 37% SWE-bench time reduction). Qwen3-Coder-Next: trajectory completion penalty + turn-level tool format penalty (token-level, not whole trajectory).

Generative Reward Models (GRM) / Rubric / Tournament : Rubric decomposes vague quality into checkable criteria; Tournament uses pairwise comparison (stable calibration). DeepSeek-V4: actor doubles as GRM, jointly optimized with rubric-guided data. MiMo: GRS (offline rubric from multi-rollout analysis: solution rubric + behavior rubric, multiplicative with binary test reward) + GAR (online groupwise grader compares all passing patches in shared workspace on 5 dimensions, hack-zeroing, advantage redistribution preserving positive advantage mass). Kimi K3: tournament with on-the-fly rubric, budget-based verbosity control (auto-loss if length > β×initial_verbosity). GLM: mixes rule-based, ORM, GRM; injects human-written style anchors. StepFun: pairwise GenRM vs fixed reference + Bradley-Terry, MetaRM penalizes spurious reasoning. Ling 2.6: bidirectional focus RM (positive+negative in one RM, dynamic weight shift from saturated rubric dimensions).

Value/Critic for token/step credit : Seed1.5-VL shared critic (RM + verifier on same scale, RM-weight init, 100-step warmup); GLM SAO skip-observation token-level GAE (significantly outperforms step-level).

Trajectory-structure credit allocation : CompactionRL cross-trajectory GAE (summary segments share final reward); DeepSeek-V4.1 DAG critical-path latency penalty; Kimi K2.5 PARL freezes subagents, only orchestrator learns, reward includes subagent completion bonus (annealed), resource constraint via Critical Steps (main agent steps + slowest subagent steps).

Open disagreement : Seed1.5-VL explicitly avoids process supervision (truncates CoT for RM); most teams (MiMo, Kimi, DeepSeek, GLM, LongCat) use sequence-level advantage broadcast, no independent PRM; Ling/MiniMax adopt process reward. "Investing grader compute for fine-grained outcome scoring (school 2) vs. process supervision are different; former is consensus, latter unresolved."

3.7 Reward Hacking

Stronger agents exploit environment: git history ground truth, verifier vulnerabilities, eval context tampering, artifact forgery, speculative compatibility. Four defense layers:

Environment hardening : remove leakage paths, network isolation, kernel hardening.

Two-tier detection + online interception : rule-based high-recall + LLM judge high-precision, intercept suspicious calls.

Verifier hardening : AST/rule matching replaces hackable logic.

Adversarial auditing + reward zeroing : dedicated hack agent probes vulnerabilities until none found.

Real cases: Qwen3-Coder-Next: agents re-add git remotes or curl/clone GitHub; heuristic intercept on tool calls containing both repo URL and git/curl/wget keywords. DeepSeek-V4.1: kernel-level exploits (XFS, AppArmor), delete binaries/filesystems; per-sandbox AppArmor + eBPF, env crash = failure + "recoil" signal. LongCat Lean4: AST parser detects 9 cheat patterns (theorem mutation, #exit, axiom / opaque, unsafe / partial); hacked model 97.9% → 27.9% after fix. Kimi K3: per-task red lines (build failure = zero, kernel task penalizes fake speedups). GLM-5.2: intercept returns dummy info (not abort) to avoid training instability. MiMo: hack agent continuous red-teaming, confirmed hack rate <2%. StepFun: zero reward for fabricated citations, overconfidence, language inconsistency; MetaRM penalizes "correct conclusion, wrong reasoning". Ling 2.6 audit: without online grading, policies learn speculative compatibility branches, over-wide exports, exception swallowing—"soft hacks" invisible to binary verifiers.

3.8 Sandbox/Environment Noise & Long-Context Management

Unique Agentic RL problem: environment as infrastructure. Two sub-problems: (1) environment noise (flaky tests, timeouts, infra crashes) corrupts rewards; (2) long-context management (compression/summarization/discard) affects task success.

Environment side: (1) Noise filtering: MiMo 8x rerun consistency, penalty module masks infra failures; Ling removes unstable instances; StepFun masks transient build failures; DeepSeek treats env crash as failure with recoil signal. (2) Strong isolation microVM: Kimi AgentENV (Firecracker, 133ms/49ms checkpoint/resume, 6.5x memory overcommit, pause/fork/snapshot, 51M+ sandboxes); DeepSeek DSec (4 tiers, custom placement engine, sub-NUMA 2500+ containers/node); Ling ASandbox (100ms start, 5000 QPS, throughput over isolation depth). (3) Massive env synthesis: Qwen 807k real PR + 851k synthetic bug Docker images; GLM 10k+ SWE, thousands Terminal, 2M+ web KG; StepFun 50k verified code envs (build success ~40%, env construction as RL task); LongCat 20+ domains, 60+ tool graphs, >95% schema-to-code success; MiniMax 100k+ scaffolds; Ling 220k Docker images via Claude Code + MCP. (4) Noise-robust training (LongCat unique): inject instruction/tool noise via automated pipeline, curriculum by τ²-Noise/Vita-Noise gap.

Context side: (5) Heuristic compression: GLM-5 keep-recent-k (k=5) + hierarchical HCM (discard all tool history >32k); LongCat hybrid (80k trigger summary + max-turn discard-all + progressive discard); Kimi K3 context management as configurable harness module; DeepSeek-V4 interleaved thinking preserves full reasoning history across user messages. (6) Context management as RL action: CompactionRL (policy generates summary when budget <10k, cross-trajectory GAE credit, SWE-bench 59.8→66.8); MiniMax models CM as explicit agent action in environment dynamics, eliminating train-inference distribution shift.

Key trade-offs: preserve vs. discard thinking history (GLM/Kimi preserve all, StepFun selective); summary vs. restart (LongCat summary, Kimi K2.5 subagent context sharding). "How to manage context is a first-class problem co-designed with training distribution, not an inference-time patch."

Cross-Team Comparison Table

Detailed 11-team alignment across 9 dimensions (architecture, sync/async, staleness, RL algorithm, train-inference mismatch, credit assignment/reward, reward hacking, environment/sandbox). Key patterns: framework base converges on Megatron+vLLM/SGLang+Ray; custom orchestration layers ubiquitous; parameter hot-swap critical (Ant AState 10s trillion-param sync vs. Slime minutes).

Six Emerging Paradigms

Sync/near-sync colocated + masking correction (Kimi K3, Hunyuan): near on-policy, clear engineering boundaries, minimal per-experiment GPUs; cost: complex memory juggling, lower throughput ceiling.

Async + partial rollout + staleness control (GLM-5, DeepSeek-V4.1, LongCat DORA, Ling 2.6): high throughput, long-tail digested; cost: precise staleness control (version cap/admission/masking), disaggregated adds weight-sync infra burden. Note: colocated/disaggregated ⟂ sync/async—freely combinable.

Sequence-level IS (GSPO family) bypasses MoE routing replay (Qwen, LongCat, Ling-2.6): optimization unit = reward unit, insensitive to token likelihood, eliminates routing replay, may skip old-logprob recompute; cost: coarse advantage granularity (needs GSPO-token for multi-turn), opposes routing-replay camp (MiMo R3, DeepSeek concat replay).

Expert-specialization + distillation replaces partial RL (DeepSeek OPD >10/>40 teachers, Kimi MOPD 9 experts×reasoning strength, MiMo MOPD2, Ling/StepFun expert-distill, GLM on-policy cross-stage distill 10+ experts in 2 days, Qwen3-Coder-Next expert distill): avoids mixed-RL negative transfer/catastrophic forgetting, low-cost merge, covers hard-to-reward domains; cost: extra distillation stage/teacher scheduling; diverges from MiniMax single-stage multi-domain RL.

Value/Critic regression vs. critic-free group-relative (Critic: GLM SAO/CompactionRL, StepFun MIS-PO, Seed PPO+shared critic+value pretrain; Critic-free: most GRPO/GSPO/CISPO). Critic enables token/step credit, single-rollout, variable-length segments, truncation bootstrap; cost: critic hard to fit long sequences, needs value pretrain/frozen attention/higher update freq—"critic is not a free baseline, it's a subsystem requiring tuning."

Environment as infrastructure: sandbox/synthesis/noise-robustness (DeepSeek DSec, Kimi AgentENV, Qwen million Docker, LongCat noise-injection RL, Ling 220k images, StepFun env-build-as-RL): elevates environment to first-class citizen, scalable synthesis + strong isolation/fault-tolerance ensures reward quality/reproducibility; cost: env synthesis/sandbox infra becomes massive engineering/compute investment (grader compute as independent scaling dimension per MiMo).

Unresolved Disagreements & Open Problems

Colocated vs. disaggregated undecided : Kimi/Hunyuan colocated, GLM disaggregated, DeepSeek-V4.1 hybrid—all produce frontier models; several teams (MiMo, Qwen, StepFun, Seed) undeclared.

No unified async staleness cap : MiMo 4, StepFun observes ~20 steps; quantitative consensus lacking.

Mismatch: system-level elimination vs. algorithmic correction : DeepSeek bitwise alignment + native FP4, MiMo/Kimi QAT shared quantization, MiniMax/Ling FP32 LM Head (elimination) vs. IcePop/KPop/DIS/MIS-PO/HisPO (algorithmic masking). Ring-1T admits IcePop hasn't achieved perfect train-inference parity; algorithmic correction not endpoint.

Routing replay: needed or not? Qwen GSPO claims "no longer needed"; MiMo insists on R3; DeepSeek-V4.1 uses concat replay.

Critic's value : Critic camp (GLM, StepFun, Seed) argues token/step credit essential; critic-free camp says group-relative baseline sufficient and cheaper. SAO shows token-level GAE beats step-level, but critic long-sequence fitting/cold-start remains hard.

Process reward worth it? Seed1.5-VL deliberately avoids CoT supervision; most teams no independent PRM; Ling/MiniMax adopt process reward. No consensus on ROI vs. complexity.

Reward hacking: perpetual arms race : "harden-audit-reharden" loop with quantifiable metrics (confirmed hack rate, AST pass rate, rollout_probs_diff). MiMo iterates until hack agent finds zero exploits; DeepSeek calls for next-gen benchmarks prioritizing detection/mitigation; MiMo ToolRep notes RL-internal penalties for tool-call repetition generalize poorly, slow, costly (20-step MixRL re-run ~$2.31M).

Environment noise & reproducibility : flaky test/infra failure/timeout handling mostly "unmentioned" or ad-hoc (MiMo 8x rerun, Ling remove unstable, LongCat inject noise); DeepSeek warns standard eval infra increasingly gameable.

Long-horizon credit assignment still open : CompactionRL cross-trajectory GAE, SAO skip-observation GAE, MiMo GAR, DeepSeek DAG critical-path—all preliminary, authors admit approximations with train-test mismatch (CompactionRL notes single-window eval gains inconsistent). Precise credit over hundreds of steps unsolved.

Conclusion

Across 11 teams' 2026 reports, Agentic RL infra has moved from "can it run" to "how to scale stably, fast, and unhackably". Async, partial rollout, train-inference consistency correction, environment-as-infrastructure, expert distillation/merging are converging into recognizable engineering paradigms, while core choices—colocated/disaggregated, sequence-level IS/routing replay, critic existence, process reward value—remain sharply contested.

Deeper tension: algorithmic purity vs. engineering reality —invest to approach clean on-policy synchronous training (bitwise alignment, strict staleness, routing replay) or embrace messy reality and absorb bias via algorithmic robustness (sequence-level IS, masking/truncation, binary acceptance). Currently the latter "embrace imperfection, algorithmic robustness as safety net" dominates, but system-level alignment persists; long-term coexistence and complementarity likely.

Whoever first cracks long-horizon credit assignment and noise-robust reproducible large-scale environment synthesis , and closes the model-harness-environment co-design loop, will seize the initiative in the next round of Agentic intelligence competition.