Reproducible Three‑Dimensional Evaluation of DeepSeek Harness on Alibaba Cloud AgentLoop

Introduction

With large language models (LLMs) moving from text generation to task execution, the runtime environment "outside the model" becomes a core issue for AI agents. The authors adopt the engineering proposition Agent = Model + Harness , where the model defines the capability ceiling and the harness (e.g., DeepSeek Harness) provides tool invocation, context management, permission control, and session logging.

Challenges of Harness Evaluation

Traditional QA benchmarks cannot measure an agent’s ability to change system state, and LLM‑as‑Judge scoring suffers from variance and leniency bias. Moreover, binary pass/fail rates hide differences in task completion quality, result legitimacy, and execution reliability.

DeepSeek Harness Architecture

DeepSeek Harness (DSH) is built on the Cordis micro‑kernel with a "everything is a plugin" philosophy. Model adapters, agent loops, session persistence, tools, security policies, and the web UI are separate packages registered to the kernel. Over 230 workspace members implement independent capabilities, enabling temporal and spatial composability as described in the paper A Programming Paradigm for Spatiotemporal Composability .

AgentLoop Platform Integration

AgentLoop provides a full‑stack observability platform for AI agents. The authors install the LoongSuite‑Pilot collector on macOS via a curl script, which injects a cordis.patch.yml into the DSH installation, enabling non‑intrusive session‑log capture. The collected Session Log contains prompts, tool schemas, model requests, tool results, permission changes, and cancellation reasons, forming a single authoritative event source for downstream evaluation.

Dataset Construction

The evaluation uses a 10‑task subset ( terminal_dataset_10.jsonl) derived from terminal‑bench 2.1. Each record specifies a self‑contained Docker image ( docker_image), an instruction ( instruction), expected outputs as pytest assertions ( test_outputs), and execution constraints ( exec_spec) such as CPU, memory, network, and wall‑clock budget.

Three‑Dimensional Deterministic Evaluators

Three orthogonal evaluators are implemented as AgentLoop AGENT + Skill units:

Outcome – measures task completion by aggregating weighted scores from verified assertions. Non‑verifiable items are excluded, and a capping rule reduces scores for partial rewards or timeout.

Compliance – a rule engine checks whether the session respects "red‑line" constraints (hard, soft, light violations) derived from the answer key. Results are passed / violated / not_verifiable and mapped to a numeric score.

Process – evaluates execution reliability using step budgets, token budgets, and productiveness thresholds, applying discounts for incomplete traces.

All evaluators read a unified payload containing task_id, the trajectory JSON, and the verifier JSON, then produce a deterministic {"score", "explanation"} JSON output.

raw = Σ(w_i·score_i) / Σ(w_i)
score = raw - 0.10·(1 - integrity)   # process discount example

Baseline Results

Using Qwen 3.7‑Plus as the underlying model, DeepSeek Harness achieved an 80% task pass rate (8/10) with a three‑dimensional average of outcome 0.74, compliance 0.98, and process 0.83, yielding an overall score of 0.85. Codex, evaluated under the same conditions, also passed 8 tasks but showed different failure patterns (e.g., extract‑elf passed for Codex but failed for DSH).

Low‑score cases are analyzed in detail: extract‑elf suffered missing output files (only 2 of 6 required), resulting in a raw score of 0.1917; hf‑model‑inference exceeded its 900 s budget, triggering a 0.50 cap despite a perfect verifier reward.

Discussion and Future Work

The authors identify four key contributions:

Evaluation objects are interchangeable; only GT reconstruction and data ingestion need to change for new benchmarks.

Ground truth is anchored to the benchmark’s answer key, avoiding subjective judgments.

Deterministic script‑based scoring eliminates LLM variance and enables fine‑grained failure attribution.

The evaluation pipeline can be run with lightweight models, reducing cost.

Future directions include scaling to full terminal‑bench 2.1 and other benchmarks (SWE‑bench, OSWorld), assetizing evaluator templates to lower the model size requirement, and closing the loop by feeding evaluation feedback back into AgentLoop for autonomous agent self‑improvement.

References

DeepSeek Harness – https://www.deepseek.com/harness/en/ and https://github.com/deepseek-ai/deepseek-harness

Terminal‑Bench 2.1 – https://www.tbench.ai/leaderboard/terminal-bench/2.1

AgentLoop – https://agentloop.console.aliyun.com/agentloop/home

LoongSuite‑Pilot – https://github.com/alibaba/loongsuite-pilot

Spatiotemporal Composability – "A Programming Paradigm for Spatiotemporal Composability"