AgenticOps Decision Models Benchmarked: 12 Models, 15K Cases, RCA Integration Insights
This article benchmarks 12 decision models on 15,788 test cases across five operational judgment types and three general tasks, finding that specialized models like Clef Pilot outperform Qwen3.8 Flash in accuracy and latency, and that integrating them at specific RCA steps improves diagnostic scores while reducing investigation time.
Background: AgenticOps and the Need for Decision Models
AgenticOps tasks — data query, inspection, and remediation — require repeated judgments: selecting objects and data, understanding evidence, choosing tools and methods, organizing context, and verifying results. The authors observed that a single RCA investigation (case F162, an RDS slow-query alert) took 10.16 minutes, invoked the diagnostic model 123 times, and scored 94/100. While the result was good, the 123 calls were spent on judgments that might be offloaded to specialized decision models, which return choices and probabilities directly without token-by-token generation.
Cost Breakdown of a Real Investigation
Across 10 recent RCA cases (from rca-benchmark), the default STAROps configuration made 1,643 diagnostic model calls totaling 214.3 minutes. Object and data selection consumed 117.3 minutes (54.7%), and context organization 41.0 minutes (19.2%). Context organization included 32 sub-investigation summaries (21.3 min), 67 observation-record calls (17.2 min), and maintenance of investigation graphs, evidence files, and pending questions.
Evaluation Design
The team built a benchmark of 15,788 scoring cases: 14,757 operational decisions covering the five judgment types, and 1,031 general decisions (sentiment recognition, model-tier selection, agent execution evaluation). Each independent judgment on the same material counts as a separate case. Twelve models were compared: four internal experimental versions (Clef Pilot 9B, Clef r9 9B, Clef SFT 27B, StartLux-style C 9B), one Bailian model (Fullset 9B), six external decision models (Perplexity Decider 27B, Decider Flash 4B, Nace.AI Drex Max 27B, Microsoft Decision 1 9B, Liquid D1 Ultra, TypeSafe Jev 1.14), and Qwen3.8 Flash as a general-purpose LLM baseline. Models were deployed on various GPUs (H20, L20, A100) or accessed via vendor APIs; all latency measurements include network and service overhead.
Key Questions
Which judgments suit decision models, and how do accuracy and latency trade off?
How do input material length and answer-conversion rules affect judgments?
When does combining a decision model with Flash (fallback on low confidence) beat either model alone?
Does integrating a decision model into the full RCA agent improve diagnostic quality and reduce investigation time?
Results: Which Judgments Benefit from Decision Models
Per-Task Accuracy
Across the five operational judgment types, every category had at least one decision model beating Flash by 1.30–7.11 percentage points. In the general tasks, decision models led in model-tier selection and sentiment recognition, while Flash remained best at agent execution evaluation. No single model dominated all categories:
Object & data selection: Clef SFT 27B 92.51% (Flash ~91%).
Information understanding: Decider 27B 82.37% (highest overall).
Tool & method selection: Clef r9 88.01% (Flash 86.71%).
Context retention (summary value): Clef SFT 27B and Decider 27B both 86.84% (Flash 57.89%).
Result verification (report evidence): Clef SFT 27B 63.03% (Flash 54.55%); but for report logic, Jev led at 74.74% (Flash 74.23%).
Fine-grained splits revealed that evidence-role classification favored Decider (78.54%) and Flash (78.40%), while report-evidence checking favored Clef SFT. This means model selection must be task-specific.
Accuracy vs. Latency on Shared Inputs
On 1,745 shared single-question inputs (mostly information understanding and tool selection), 11 decision models had P95 latency below Flash's 5.839 s. Clef Pilot achieved 82.81% accuracy (+2.52 pp over Flash) with P95 1.150 s (80.3% faster). Among 27B models, Clef SFT 27B reached 83.21% accuracy but P95 1.967 s; the extra 7 correct answers came mainly from sentiment recognition, while operational tasks slightly regressed. External services showed Liquid (80.40%, 2.193 s) and Jev (80.52%, 4.284 s) with similar accuracy but very different latency. Fullset was fastest (P95 0.251 s) but accuracy dropped to 74.90% (94 fewer correct than Flash).
Implementation Differences Within Same Scale
Among 9B models (all Qwen3.5-9B base), Clef Pilot excelled at object selection (91.38% vs StartLux 82.19%), while StartLux led in result verification (77.53% vs Pilot 72.10%). Among 27B models (Qwen3.8-27B base), Clef SFT led in object selection (92.51%), tool selection (85.74%), and result verification (79.75%), but Decider 27B led in information understanding (82.37% vs 79.32%). Drex Max and Decider tied on context retention (~85.2%). Input organization also differed: Decider jointly encodes material with multiple questions; Drex Max encodes per question; StartLux uses 26-letter readout heads. These architectural choices affect multi-question efficiency and should be tested under production-like batching.
Interpreting Scores and Using Model Outputs
Material Length Effects
Across 10 models, accuracy peaked in the 2k–8k character bucket and fell at both shorter and longer lengths. However, task-level analysis showed opposite trends: root-cause entity identification improved with length (Pilot 81.68% → 93.02%), while fault-type classification degraded (Pilot 70.76% → 47.37%). The takeaway: instead of targeting a generic length, teams should add missing evidence or remove irrelevant content per task and measure the impact.
Answer Conversion Rules Change Rankings
Models return continuous scores and per-option probabilities. Converting continuous scores to discrete bins (e.g., rounding) altered rankings dramatically. On 114 summary-value questions, StartLux jumped from 46 to 83 correct (surpassing Flash's 66) when using top-probability bin instead of rounded score; Liquid dropped from 105 to 89. The same response yielded different decisions purely due to post-processing. The authors recommend fixing the conversion rule (e.g., always use highest-probability bin for ordinal tasks) during evaluation and production.
Error Direction Matters for Downstream Actions
On summary-value questions, a baseline of always picking the majority bin (tier 3) yields 78.07% accuracy. Only Clef SFT 27B, Decider 27B, and Drex Max 27B exceeded this. Pilot's overall win (81 vs 66 correct) masked a trade-off: it gained 26 more tier-3 correct but lost 11 on tier-2. Jev's 72 errors were all underestimates (40 tier-3→2, 15 tier-3→1), risking deletion of useful evidence. Liquid's 25 errors were mostly overestimates (23 tier-3→4), risking premature termination of context reading. Error direction must be analyzed against the action it triggers.
Model Combination with Fallback
Using 14,245 operational questions, each decision model was paired with Flash: high-confidence answers kept, low-confidence routed to Flash. Thresholds were selected via 5-fold cross-validation per model. Results:
49 of 55 model–task pairs improved over the decision model alone; 38 pairs beat the stronger of the two single models.
Example: Clef SFT 27B on tool selection (85.74%) + Flash (86.71%) with 20.10% fallback reached 88.98% (+2.27 pp over Flash alone).
Conversely, Drex Max 27B on context judgment (85.25%) fell to 84.32% with fallback; Clef r9 on tool selection (88.01%) fell to 86.39%. For these, standalone use is better.
Per-task thresholds sometimes increased fallback volume without proportional gains (e.g., Clef SFT 27B added 477 fallbacks for 46 more correct). The cost–benefit of finer-grained routing must be weighed against actual latency and token overhead.
RCA Agent Integration Results
Ten RCA cases were run end-to-end with the main investigation model fixed (DeepSeek v4.1 Flash) and a decision model inserted at one of five positions: B (object & data source), C (task dispatch), D (progress & replanning), E (context management), F (root-cause verification). Three decision models were tested: Microsoft Decision 1, Clef Pilot, Clef 27B (Clef SFT). Baseline A: 81.10 score, 21.49 min, 12.892M tokens.
Position Decision 1 Clef Pilot Clef 27B B Object & Data 77.50 / 21.02 / 12.676 73.00 / 21.71 / 11.777 72.35 / 22.36 / 13.249 C Task Dispatch 87.50 / 23.56 / 12.439 80.00 / 21.19 / 12.499 76.50 / 23.19 / 12.575 D Progress & Replan 81.20 / 18.42 / 13.297 74.60 / 28.63 / 12.511 88.30 / 22.25 / 12.876 E Context Mgmt 71.23 / 31.65 / 14.455 82.70 / 21.66 / 13.854 77.20 / 20.93 / 13.464 F Root-Cause Verify 76.45 / 18.91 / 13.707 83.50 / 20.63 / 11.868 71.80 / 27.09 / 12.845Key findings:
Position matters more than model: Decision 1 at C (+6.40 pts, +9.6% time), Clef 27B at D (+7.20 pts, +3.5% time), Clef Pilot at F (+2.40 pts, -4.0% time).
Object selection (B) saw no improvement over baseline; keep original flow there.
Latency and token usage do not always correlate: Decision 1 at D reduced time 14.3% but increased tokens 3.1%; Pilot at E added 1.6 pts with flat time but +7.5% tokens.
Per-case analysis showed concrete improvements: Decision 1 C linked memory growth with traffic symmetry in largeValue case and identified the maintenance pod triggering sync deletion in bigKey case. Clef 27B D's 7.2-point average gain came largely from largeValue (+6.5 pts). The cache-list-growth case remained weak across configs, stuck at linking List/LPUSH/client-IP to specific Pod/Job.
Related Work
The article references recent preprints on decision-model capabilities (Jev vs. generative judges, AbductionBench), efficiency (TrialJev shows 2.45–2.70× speedup over generation), calibration (Beyond Ranking separates ranking quality from probability reliability), and input organization (missing-candidate effects, cross-primitive consistency). These align with the experimental themes of this work.
Conclusion
High-frequency operational judgments (object selection, evidence understanding, action choice, context management, result verification) are ready for offloading to specialized decision models. Clef Pilot already delivers higher accuracy and lower latency on shared inputs.
Offline leaderboards are insufficient; integration design (material preparation, answer conversion, fallback policy) determines whether single-step gains survive in the full agent. The winning configurations in RCA were Decision 1 at task dispatch, Clef 27B at progress judgment, and Clef Pilot at root-cause verification.
Improvements must be traced to concrete fault cases. The cache-list-growth case exposes a clear next target: associate List/LPUSH/client-IP with the responsible Pod/Job. This trajectory — offline task → full-agent integration → per-case gap → next experiment — drives the team's Agent RSI loop.
Signed-in readers can open the original source through BestHub's protected redirect.
This article has been distilled and summarized from source material, then republished for learning and reference. If you believe it infringes your rights, please contactand we will review it promptly.
How this landed with the community
Was this worth your time?
0 Comments
Thoughtful readers leave field notes, pushback, and hard-won operational detail here.
