DeepSeek Harness Real-World Test: What the Non-Model Half Actually Delivers

Setup and entry points

Installation uses npx @deepseek-ai/dsh web (Node ≥22.19.0). Four entry points run successfully: Web UI (creates local workspaces and sessions), Headless (one‑off task, prints final reply), Python SDK (bundles runtime, no Node required), and --dump-config (prints the assembled plugin tree). The default profile loads 129 plugin entries for Web and 81 for Headless; the default model is DeepSeek V4‑Flash with workspace‑write + ask permissions.

Trajectory logging

DSH records every turn, step, tool call, token usage, and cache hit in a zstd‑compressed JSONL stream. Each event has a uniform envelope, e.g.

{"type":"tool/call","seq":31,"time":1786632922876,"data":{}}

. A simple write‑file task generated 61 events, including turn/start, three step/start, user messages, request headers, reasoning chunks, two tool calls, and turn/end. This granularity answers three questions: (1) what the model saw, (2) which tool was invoked at each step, and (3) how token and cache usage evolved.

Plugins vs. presets

Plugins add new capabilities; presets select which capabilities an agent can see. An example replaces the generic bash tool with a sqlcmd tool, creating a data‑agent that only reads, edits, and writes database results. The design encourages narrowing the agent’s focus rather than continuously adding tools, which would increase token consumption and model confusion.

Token overhead in long‑running tasks

A minimal Headless task that only replies “PONG” still consumes ~13,467 input tokens because the default system prompt, tool descriptions, repository rules, and skill summaries are injected into every request. In a large repository the .git root caused injection of AGENTS.md and CLAUDE.md, plus 27 skill summaries, inflating cache‑read token count.

Side‑by‑side comparison with Kimi Code

Both DSH and Kimi Code are open‑source, support the same Kimi K3 model, and use identical prompts and hidden acceptance criteria. Two benchmark tasks were run once per harness.

Dependency planning – DSH minimal: 112.2 s, 9 steps, 11 tool calls; Kimi Code: 48.9 s, 4 assistant messages, 5 tool calls.

Session projection – DSH minimal: 111.3 s, 7 steps, 7 tool calls; Kimi Code: 123.1 s, 5 assistant messages, 6 tool calls.

Both harnesses achieved perfect correctness (15/15 acceptance). Speed varied per task with no consistent winner. Kimi Code batches reads (README, source, tests) and performs fewer tool calls; DSH fragments the workflow into more granular steps, producing richer session logs for reproducibility.

Token usage recorded by DSH: 7,593 input / 5,082 cache‑read tokens for dependency planning and 22,515 input / 18,632 cache‑read tokens for session projection. Kimi Code’s stream‑json output does not expose token usage.

Full‑stack game generation

Both harnesses generated a “jump‑the‑block” web game using the same V4‑Pro model, prompt, and acceptance criteria (visual output, backend leaderboard, automated tests). Results:

Kimi Code produced a 2.5D full‑screen stage with polished visuals.

DSH produced a 2D canvas with a side panel for the leaderboard and instructions.

Execution times were comparable (≈21 min for Kimi Code, ≈25 min for DSH). Both required post‑generation fixes (Kimi Code’s updateHUD call, DSH’s canvas size bug), demonstrating that the same model can yield distinct products depending on the harness.

Architecture deep dive

DSH is a TypeScript monorepo built on the Cordis plugin framework. Core packages: agent-loop – turn/step driver. session – append‑only event log (zstd‑compressed JSONL). tools – tool registration and execution. system-prompt – prompt fragment assembly.

Presets assemble a plugin tree for each session; profiles configure the process‑level plugins. The vendor/cordis layer manages lifecycle states ( PENDING → ACTIVE) and disposes effects via ctx.effect().

System prompts are reconstructed from logs before each request; a mismatch triggers a log‑reconstruction desync error, guaranteeing that the model only sees what is recorded.

File system abstraction separates FileSystem, opaque targets, and versioning, allowing local or sandboxed providers to be swapped without changing higher‑level tools. Operations such as createIfAbsent and replaceIfVersion protect against stale writes.

Program‑to‑Code (PTC) lets the model generate TypeScript that orchestrates multiple tool calls, executed in a worker_threads sandbox. This reduces round‑trip latency but requires additional isolation.

Plugin system (Cordis)

Plugin entry points reside in vendor/cordis/src/context.ts, fiber.ts, and service.ts. Plugins declare dependencies via inject. When a required service is missing the plugin stays in PENDING; once the service appears it transitions to ACTIVE. On provider exit, dependent components are unloaded and their effects disposed with ctx.effect(). This mechanism governs the lifecycle of the Agent Loop, which is itself a Cordis plugin.

Profile and preset composition

Profiles start from an empty bundle, then apply patches, home directories, and command‑line overrides. The final plugin tree can be printed with dsh --profile web --dump-config, ensuring configuration consistency between dump and runtime.

Presets decide which tools, prompt fragments, and projection units a session sees. The same process can run multiple presets (standard, minimal, creative) concurrently, enabling domain‑specific agents such as the Data Agent that swaps bash for sqlcmd.

Invariant check for request reconstruction

Before each request the runtime derives the expected message list via session.deriveMessages() and compares it to the actual request options. If they differ, a log‑reconstruction desync error is thrown:

const expected = session.deriveMessages()
if (JSON.stringify(options.messages) !== JSON.stringify(expected)) {
  fail('log-reconstruction desync')
}

This guarantees that the model’s view is fully captured in the session log and that token consumption can be audited.

Capability seam (file system & sandbox)

The file‑system API defines FileSystem, opaque targets, and version objects. Implementations include a local provider ( fs-local) and a sandboxed provider ( fs-sandbox). Tools such as tool‑str‑replace‑editor consume the abstracted file system, so swapping the underlying provider does not require tool changes. Write operations support createIfAbsent and replaceIfVersion to avoid stale overwrites and ensure concurrency safety.

Multi‑model support

DSH ships with a DeepSeek direct adapter and a pi‑ai multi‑provider adapter. Local smoke tests exercised Kimi K3, GPT‑5.6 Sol, and Claude Opus 4.8 for simple file write/read and tool invocation, confirming that provider selection, OpenAI‑compatible messaging, SSE streaming, and tool protocols function correctly.

Key observations

The same model and prompt can produce markedly different products: Kimi Code favors a full‑screen 2.5D experience, while DSH yields a 2D canvas with integrated leaderboard.

Both harnesses can handle 20‑minute, multi‑step projects with backend components and automated tests; DSH’s fixed step granularity and seeded PRNG aid reproducibility, whereas Kimi Code’s richer default toolset accelerates execution.

Model‑generated tests may pass while real‑browser execution fails (e.g., DSH canvas size bug, Kimi Code CSS hidden issue). The full trajectory log enables post‑mortem debugging, but final validation still requires external execution and human review.

Repository layout

deepseek-harness/
├── apps/cli + apps/web          # CLI and Web entry points
├── packages/boot + bundle       # Profile and plugin‑tree assembly
├── packages/core/
│   ├── agent-loop               # Turn/Step driver
│   ├── session                  # Append‑only event log
│   ├── tools                    # Tool registration & execution
│   └── system-prompt            # Prompt fragment assembly
├── packages/preset              # Per‑session agent composition
├── packages/llm                 # DeepSeek & multi‑provider adapters
├── packages/fs + sandbox        # Swappable file‑system providers
├── packages/code-runtime        # PTC / Code Mode implementation
└── vendor/cordis                # Plugin lifecycle framework

Practical takeaways

Agent‑infrastructure engineers can explore DSH for its transparent loop, session, provider, tool, and UI layers.

Domain‑specific teams can start from the minimal or standard preset, prune tools, and swap providers to evaluate benefits.

Model‑evaluation researchers should lock model version, inference tier, endpoint, and external acceptance criteria while using the minimal composition.

Production deployments must handle plugin provenance, configuration diffs, credential boundaries, Windows validation, log retention, and data egress.

Open questions for future harness designs

Can the runtime print the entire loaded plugin tree with a single command?

Is it possible to fully reconstruct what the model saw from the logs?

When swapping file systems, sandboxes, or model providers, how many tools need adjustment?

References

https://github.com/deepseek-ai/deepseek-harness

https://github.com/deepseek-ai/deepseek-harness/blob/master/docs/architecture.zh.md

https://github.com/deepseek-ai/deepseek-harness/blob/master/docs/user/guide/python-sdk.md

https://github.com/cordiverse/cordis

https://github.com/MoonshotAI/kimi-code

https://arxiv.org/html/2605.26144 (VISTA benchmark)