Why DeepSeek V4 Pro’s 87.9 Score Signals Agent Benchmarks Moving from Model to System

DeepSeek V4 Pro‑0813 achieved an official 87.9 score on Terminal‑Bench 2.1. The benchmark note specifies that the Code Agent task ran in DeepSeek Harness Minimal Mode with max reasoning effort, temperature 1.0 and top_p 0.95, and warns that results may differ under other frameworks.

Vals AI’s independent evaluation of the same model reported a 54.68 % pass@1 score on the same benchmark, using Terminus 2 harness with identical configuration. The discrepancy is not a simple performance gap but stems from different Agent Frameworks and experimental settings.

The article explains that traditional model‑only benchmarks follow a Prompt → Model → Answer → Score pipeline, whereas Agent benchmarks require continuous interaction with a sandboxed environment, executing commands, handling failures, and updating context. Consequently, the harness (runtime system) becomes a critical component of the evaluation.

DeepSeek Harness offers four modes—Standard, Code, Minimal, Creator. Minimal Mode retains only a persistent bash and a string‑replace editor, explicitly intended for model benchmarking. By reducing auxiliary tools, the harness isolates model capability, but it also means the benchmark score reflects both model and harness behavior.

Three layers of influence are identified: (1) Context handling—what information is retained or discarded between steps; (2) Tool set—availability and design of shells, editors, search, browsers, etc.; (3) Agent loop mechanics—how errors are fed back, when the next model call is triggered, and the imposed reasoning budget.

DeepSeek’s logging system records system prompts, reasoning steps, tool calls, results, sub‑agent scheduling, and context injections, enabling reproducibility and traceability of the entire Agent execution.

The shift from “model benchmark” to “system benchmark” means that high Agent scores must be interpreted with full disclosure of Model, Harness, Tool Set, Reasoning Effort, Context Strategy, step limits, token budget, environment, and cost. Only with these details can scores be meaningfully compared.