DeepSeek Harness Outperforms Claude Code on Bug Hunt Bench with V4.1 Flash 1017

Bug Hunt Bench Evaluation: DSH vs. Claude Code

Independent developer Paweł Huryn published a third-party benchmark called Bug Hunt Bench . The test uses two unrelated production codebases — a ~28K-line TypeScript VS Code extension and a ~60K-line React/TypeScript LMS system — with 105 hidden bugs implanted. The environment is fully offline, Git history is wiped, and the test suite stays green, forcing the model to read the code like an architect. An independent judge blindly evaluates diffs against secret answers, counting only truly fixed implanted bugs.

Scores (DeepSeek-V4.1-Flash-1017, same API, same reasoning effort)

Max tier : Claude Code 39.1 (7-run avg) vs. DSH 42.7 (7-run avg) → +3.6

High tier : Claude Code 36.4 (5-run avg) vs. DSH 39.2 (5-run avg) → +2.8

Low tier : Claude Code 34.4 (5-run avg) vs. DSH 34.8 (5-run avg) → +0.4

Across the three tiers, DSH averages 2–3 points higher . At Max tier, DSH's 42.7 exceeds the scores of flagship models running inside Claude Code: Claude Opus 5.5 (41.7) and Muse Wonder (42.0) , trailing only GPT-6 Astra (45.0) .

Cost Analysis: The Price of Higher Scores

Paweł's published token receipts reveal the trade-off. In Max tier, aggregated over both projects:

Per-task actual bill : Claude Code $1.01 → DSH $1.84 ( +82.2% )

Output tokens : 410K → 854K ( +108.1% )

New input tokens : 1.035M → 2.549M ( +146.2% )

Total tokens : 208.957M → 319.912M ( +53.1% )

Interaction turns : 671 → 804 ( +19.8% )

Extra fixes (non-scored) : 22.9 → 44.0 ( +92.1% )

Why DSH Wins: Subagents and Full-Permission Mode

Claude Code follows a philosophy of extreme restraint and single-thread focus : it aggressively trims context and output length, prefers conservative early stops on ambiguous issues, yielding low per-task cost (~$1).

DSH (v0.2.1-rc.2) defaults to Subagents (child-agent mechanism) and runs in full-permission mode. When facing tough bugs, it spawns sub-tasks, self-reflects, retries, and launches deep investigations. Output tokens double, total bill rises 82%, but the extra compute is not wasted: besides the 3.6 additional scored bugs, DSH also fixed 44 non-scored historical defects — twice as many as Claude Code.

This "relentless hound" debugging style is often the key to breaking deadlocks in large, complex engineering troubleshooting.

Paradigm Shift: Evaluation Unit Moves from "Model" to "Model × Harness"

The 1017 breakthrough on DSH validates an October 2024 ArXiv paper: "Finding the Right Fit: Model–Harness Interactions across Agent Tasks" . The study ran 6,200+ trajectories across multiple top models and four mainstream harnesses and found:

Swapping the harness alone can completely flip leaderboard rankings for the same model.

No universal "champion combination" exists independent of the specific task.

The model is like a high-power engine; the harness (prompting, tool protocols, subagent scheduling, context trimming) is the chassis and transmission.

Deployment Strategy Based on Task Profile

Daily light coding, single-script fixes, strict budget control : Continue preferring Claude Code + 1017 . Per-task cost stays lower, token bloat is tightly contained, cost-effectiveness maximized.

Complex project refactoring, long-horizon deep-water bug hunting, multi-module coordination : Embrace DSH (v0.2.1-rc.2 or later) without hesitation, paired with the DSH Desktop client for trajectory observation. Although per-task bill rises ~80%, against DeepSeek's already extremely cheap API pricing, the small extra cost buys deep-water debugging capability that is well worth it.

Conclusion

LLM development has moved past pure base-model parameter competition. Agent-era competition has evolved into a system battle of deep coupling between model capability and scaffolding engineering .

Code example

DeepSeek-V4.1-Flash-1017