A3 Framework: 14B CLI Agent Beats 235B Model via Structured Action Credit Assignment

Researchers from Fudan University, Shanghai Jiao Tong University, and Shanghai AI Laboratory propose A3 (Action Advantage Assignment), a reinforcement learning framework for command-line interface (CLI) agents that exploits the programmable structure of shell commands for fine-grained credit assignment. The paper Learning CLI Agents with Structured Action Credit under Selective Observation has been accepted at NeurIPS 2026.

Core Problem: Sparse Reward in Multi-turn CLI Interaction

CLI agents like Claude Code and Codex execute sequences of shell commands, modifying files and correcting strategies based on feedback. The final task success provides a trajectory-level reward, but individual actions — retrieval, transformation, editing, verification — play distinct roles. Applying a single trajectory reward uniformly across all steps fails to leverage structural differences among actions. Prior agentic RL methods refine advantage estimation via repeated-state anchoring or observation grouping, but CLI environments exhibit high diversity in file states and text observations; different sampled trajectories for the same task often produce different execution histories, weakening the applicability of repeated states as a normalization basis.

A3: Three-Level Advantage from AST Signatures

A3 parses Bash actions into abstract syntax trees (ASTs) using Tree-sitter. Through deterministic traversal, it extracts control structures, normalized command verbs, and normalized literals to form an AST signature, then measures pairwise differences with normalized edit distance. This representation enables unified comparison of single-step actions and multi-step action sub-chains. Within a group of trajectories sampled for the same task prompt, A3 constructs three complementary advantage components:

Trajectory-level base advantage (inherited from GSPO): measures the overall performance of the current trajectory relative to other trajectories in the same group, preserving task-level outcome supervision.

Step-level action sub-chain residual : at each turn, groups trajectories by AST signature distance of action sub-chains of varying lengths. For each group, computes the reward residual of the current trajectory against the leave-one-out mean reward of other trajectories, then aggregates residuals across sub-chain lengths with weighting. This links the final reward to local action structure: trajectories executing similar command fragments form comparable statistical sets, and multiple fragment lengths provide behavioral signals at different time scales.

Tree-structure advantage : builds an abstract interaction history from the sequence of sub-chain group labels. Uses time-weighted Hamming distance on history labels to cluster similar histories into abstract states, then groups current action branches. Abstract state value is estimated by the mean reward of trajectories reaching that state; branch value by the mean reward of trajectories taking that branch. Their difference forms a local branch-value delta, discounted and accumulated along the trajectory into a tree-structure advantage. For groups with few samples, a decay based on group size reduces the signal's influence.

The three components undergo scale alignment, weighted summation, and bounded transformation to produce per-turn advantages for sequence-level policy optimization. The entire advantage construction reuses existing sampled trajectories' rewards and action information — no additional value-model training and no external LLM step-wise judging required.

Figure 1: Overview of A3 and σ-Reveal. Left: AST-based action comparison. Top: workspace observation selection. Bottom: fusion of trajectory, step, and tree-structure advantage.

σ-Reveal: Budget-Constrained Workspace Observation Selection

Complementing training-time credit assignment, σ-Reveal constructs task-relevant initial workspace observations at inference time. It formulates context selection as a budget-constrained optimization on the file tree: choose a node set that maximizes cumulative relevance scores while preserving directory dependencies, subject to a context presentation cost budget. This provides the policy with a lightweight workspace prior.

ShellOps and ShellOps-Pro: Executable Workspace Benchmarks

CLI agent capability is reflected in actual workspace operations. In cross-file tasks, a correct final answer does not guarantee completion: whether all required files are updated and generated content satisfies constraints equally determines success. ShellOps organizes information retrieval, cross-file aggregation, and file editing into executable workspace tasks, each with an initial file environment and reference solution. Tasks evaluate text output, file modifications, and their joint correctness, so every interaction is verifiable by execution results.

ShellOps (standard) : 1,624 simple tasks for basic capability learning.

ShellOps-Pro : 150 challenging out-of-distribution tasks for complex-task generalization. Contains 4,063 files total, averaging 27.1 files per task, covering configuration, structured data, logs, code, and text. This setting extends evaluation to more complex file organization and operation demands, testing transfer of retrieval and execution skills.

Evaluation distinguishes String (text result), Files (file state), and Hybrid (joint correctness) task types. Additional benchmarks: AgentBench-OS, AgentBench-DB, DataBench, EHRCon, TableBench — covering OS interaction, database operations, table QA, and factual consistency.

Figure 2: ShellOps data distribution — file types and counts, task content composition, file size distribution, and per-task file count and total size.

Figure 3: ShellOps-Pro data distribution. Compared to ShellOps, tasks involve more files and larger workspaces, demanding higher cross-file retrieval and multi-step operation capabilities.

Figure 4: ShellOps task structure and execution flow. Tasks consist of natural language instruction, initial workspace, reference solution, and target output, covering search, aggregation, editing, and mixed operations.

Experimental Results

Main experiments use Qwen3-14B as the policy model. A3, GSPO, GiGPO, HGPO, and RetroAgent are compared under identical training configurations; ReACT, LATS, and rStar serve as agent reasoning baselines.

Figure 5: Exact match rates (%) across task types. Vanilla = standard observation; σ-Reveal = filtered workspace observation at inference.

A3 leads all baselines on ShellOps three task types. Combined with σ-Reveal, Hybrid EM jumps from the best baseline's 11.3% to 24.6% (over 2× improvement); ShellOps Pass@5 nearly doubles as well.

Figure 6: Out-of-distribution generalization on ShellOps-Pro. A3 + σ-Reveal enables Qwen3-14B to surpass Qwen3-235B-A22B in 10-turn interaction.

On the more complex ShellOps-Pro, the 14B model without additional training outperforms Qwen3-235B-A22B (≈17× total parameters): 31.1% EM vs. 28.3%.

Figure 7: Training dynamics of Qwen3-14B — task success rate, answer correctness reward, policy entropy, and KL divergence.

Training curves show sustained improvement in task success rate and answer reward with relatively stable KL, reflecting stable policy optimization dynamics.

Ablation and Computational Cost

Removing step-level and tree-structure credit assignment drops Hybrid EM from 21.9% to 10.9% (nearly halved), confirming the importance of fine-grained signals. Despite richer signals, advantage computation costs only 0.023 seconds per batch, on par with GSPO.

Figure 8: Advantage composition and accuracy vs. compute analysis. Left: fused signal magnitude and proportion. Right: ShellOps macro-average EM vs. advantage computation cost.

σ-Reveal's cost appears at inference: average per-turn input tokens rise from ~0.99K to ~1.11K, while average interaction rounds decrease from 4.33 to 4.18, illustrating the trade-off between initial workspace evidence, context overhead, and interaction length.

Open Resources

Paper: https://arxiv.org/abs/2605.08013 Code: https://github.com/Hoyant-Su/Agentic-RL-A3 Dataset: https://huggingface.co/datasets/Hoyant-Su/ShellOps Model:

https://huggingface.co/Hoyant-Su/A3-Qwen3-14B

Conclusion

A3 demonstrates an alternative path to agent capability leaps: enabling models to learn more thoroughly from interaction processes. By mining credit-assignment signals in command structure, a 14B model doubles accuracy on complex tasks and surpasses a 235B model on out-of-distribution evaluation, showing the potential for training methods to bridge parameter-scale gaps. This result expands the imagination space for Agentic RL: every executed operation contains learnable signals worth exploiting. Converting these signals into decision-making ability will be a key direction for advancing agents toward complex, long-horizon tasks. A3, together with the open ShellOps datasets, provides a methodological and experimental foundation for this direction.