How Kimi, Cursor, and Chroma Use Reinforcement Learning to Train Agent Models

Kimi, Cursor, and Chroma: RL‑Based Agent Model Training

The author reviews three recent technical reports—Moonshot AI’s Kimi K2.5 paper, Cursor’s Composer 2 report, and Chroma’s Context‑1 documentation—each proposing a distinct reinforcement‑learning (RL) approach for training intelligent agents.

Kimi K2.5 – Agent Swarm and Parallel Orchestration

Kimi K2.5 is a multimodal 1 trillion‑parameter MoE model that introduces an Agent Swarm framework, enabling the model to decompose tasks into parallel sub‑agents. Parallelization is learned via RL rather than hard‑coded.

The architecture defines two roles:

Orchestrator (trainable) : decides when to create sub‑agents, assigns tasks, and aggregates results using the tools create_subagent and assign_task.

Sub‑agents (frozen) : execute assigned sub‑tasks; their trajectories are treated as environment observations and are not optimized.

This decoupling addresses the credit‑assignment problem by freezing sub‑agents so that only the orchestrator’s coordination logic is optimized.

PARL Reward Design

PARL combines three reward components:

Performance reward r_perf : primary signal indicating task success.

Parallel reward r_parallel : encourages spawning sub‑agents and avoids "serial collapse" where the orchestrator falls back to a single agent.

Finish reward r_finish : rewards completion of sub‑tasks and prevents "spurious parallelism" (creating many useless sub‑agents just to gain r_parallel).

Auxiliary reward coefficients are annealed to zero so the final policy is purely performance‑driven.

Inference Procedure

Orchestrator analyses the task and identifies a sub‑task structure.

It creates sub‑agents with create_subagent and specific instructions.

It assigns tasks to sub‑agents via assign_task; each sub‑agent runs with an independent context window.

Orchestrator collects results, synthesises a final answer, or repeats the process.

Parallel execution is not hard‑coded; simple tasks remain sequential, while complex multi‑source research tasks trigger many parallel agents. Training data deliberately emphasize "wide search" or "deep search" scenarios to make parallelism advantageous.

Results: Agent Swarm reduces inference latency up to 4.5× and improves accuracy, achieving 78.4 % on BrowseComp (single‑agent baseline 60.6 %) and raising WideSearch item‑level F1 from 72.8 % to 79.0 %.

Cursor Composer 2 – Real‑Time RL for Agent Programming

Composer 2 is Cursor’s self‑programming agent that can read/write files, execute shell commands, search codebases, and browse the web. It is trained within the same production Cursor framework, using identical tools, prompts, system messages, and file contexts.

Key components:

Training : fully asynchronous architecture built on Ray and PyTorch.

Environment : each rollout runs in an isolated Firecracker VM (Anyrun), supporting full development environments and allowing checkpointing.

Inference : distributed RL inference with Fireworks AI; weights are incrementally compressed to S3 and sharded across ranks, enabling on‑policy weight updates mid‑rollout.

Evaluation : fixed production backend and Cursor client replica ensure evaluation matches user experience.

Composer 2 employs a self‑summarization mechanism: during long rollouts, intermediate summaries are concatenated, and the final reward is applied to all tokens, encouraging concise yet informative summaries and penalising loss of critical context.

Real‑Time RL Loop

Collect billions of tokens from live user interactions with checkpoints.

Distil user feedback (e.g., further edits, satisfaction) into reward signals.

Train on these signals to produce updated checkpoints.

Validate against regression with CursorBench.

Deploy if checks pass.

The entire loop takes roughly five hours and can be run multiple times per day, keeping data close to on‑policy.

Chroma Context‑1 – Self‑Editing Search Agent

Context‑1 is a 20 billion‑parameter agent focused on retrieving documents rather than answering questions. Its core innovation is self‑editing context : the model learns to discard irrelevant retrieved documents, freeing context space for subsequent search.

Synthetic Data Pipeline

Chroma builds a synthetic benchmark covering web, finance (SEC filings), law (USPTO patents), and email (Enron) domains. Each task follows a five‑step pipeline: collect factual documents, generate indirect clues and questions, verify verbatim citations, add distractor documents, and optionally chain tasks into multi‑hop questions.

Agent Tool Framework

search_corpus(query)

: hybrid BM25 + dense retrieval with RRF fusion. grep_corpus(pattern): regex search. read_document(doc_id): fetch a specific document chunk. prune_chunks(chunk_ids): remove irrelevant chunks from context.

The framework enforces a fixed token budget (e.g., 32 k tokens). When a soft threshold is crossed, a pruning hint is issued; crossing the hard limit locks all tools except prune_chunks, forcing the model to prune or terminate.

Deduplication is handled at the framework level by tracking seen chunk IDs and automatically excluding them from future searches.

Training: SFT Warm‑up + RL (CISPO)

SFT warm‑up uses Kimi K2.5 to generate trajectories, retaining high‑recall ones and sampling a small fraction of low‑recall or zero‑recall examples as negatives.

RL employs CISPO (Clipped Importance‑Sampled Policy Optimization), a GRPO variant. Each step samples 128 queries × 8 rollouts = 1024 trajectories; rollouts with identical group rewards are discarded to avoid zero‑gradient signals.

Reward Design

Result reward : F‑beta score, with early beta set high (recall weighted 16× precision) to prioritise "no miss" over "no excess".

Process reward : trajectory recall, rewarding the model for seeing relevant documents even if later pruned.

Final answer reward : +1.0 for retrieving a document chunk containing the actual answer.

Penalties : repeated‑prune penalty (discourages pruning one document at a time) and round‑penalty (prevents diminishing‑return search loops).

Common Themes Across the Three Systems

Training in the deployment environment : all teams invest heavily to align training tools, prompts, and execution environments with production, minimizing train‑to‑deploy gaps.

Context management is a first‑class concern : Cursor uses self‑summarization, Kimi employs parallel sub‑agent slicing, and Chroma adopts active pruning; all treat the limited context window as a resource to be actively managed.

Reward design is an iterative process : each team observes reward‑hacking behaviours (e.g., Cursor’s useless tool calls, Kimi’s serial collapse, Chroma’s premature search termination), analyses the incentive mis‑alignment, and adds targeted rewards or penalties.

Public benchmarks are insufficient : Cursor builds CursorBench from real user sessions, Chroma creates multi‑domain synthetic benchmarks, and Kimi combines public and internal evaluations, highlighting the need for vertical‑specific testing.

Specialised, smaller models can rival larger ones : Chroma’s 20 B model matches frontier large models on retrieval tasks with ten‑fold speed gains; Composer 2 achieves a better cost‑accuracy trade‑off than larger API models, showing domain‑focused RL can bridge parameter‑scale gaps.

Original source: https://www.philschmid.de/kimi-composer-context