From Bug Fixes to Completed Work: Insights from Tencent’s WorkBuddy Bench

Fake completion in software development

Agents often fix a failing test and turn the target case green, yet the change is not handed over properly. Subsequent integration may reveal missing time‑zone coverage, absent compatibility fields, or outdated configuration references. The bug is fixed, but the work is not truly delivered.

Four‑layer model of completion

Answer : a paragraph, plan, or code snippet. Missing : entry into the real work area.

Action : file changes or tool invocations. Missing : certainty that the result is complete.

Delivery : patch, web page, report, or PoC. Missing : verification of state and constraints.

Completion : deliverable is usable, state is consistent, evidence is auditable. Missing : readiness for hand‑off, release, or next step.

Code alone only shows that an action finished; additional checks such as CI results, interface compatibility, data migration, and release gates are required to deem the work complete.

Task packaging in WorkBuddy Bench

task/
├── instruction.md   # natural‑language request
├── task.toml        # category, difficulty, resources, timeout
├── environment/     # Docker workspace visible to the agent
├── tests/           # evaluation assets executed after the task
└── gold.patch       # optional diagnostic reference for code tasks

This layout fixes several boundaries: the agent’s starting version and files, visible assets, allowed tools and network, output locations, and the post‑run verifier.

Benchmark objects vs. real‑world artifacts

instruction.md

: requirement card or ticket – defines goals and constraints. task.toml: execution strategy and resource limits – defines runtime length and allowed resources. environment/: fixed version of dev/test environment – defines start point and reproducibility. tests/: CI, acceptance rules, quality gates – defines how to prove the change is valid. results/: build artifacts, logs, release evidence – defines how to audit and compare later.

Four work‑space tracks

Code – 80 tasks covering bug fixes, feature pipelines, testing, etc. Average scores: bug_fix 0.47, api_contract 0.47, feature_pipeline 0.94, testing 0.88. Typical failure modes include endless edits to test files or wandering in large repositories.

Web – 70 tasks requiring the agent to leave a runnable front‑end artifact. Success depends on persisting pages, handling state changes, and preserving data after refresh.

Office – 50 tasks with mixed outputs (xlsx, csv, PDF, markdown). Completion demands consistency across generated files, numbers, and status files; rule‑weight ranges from 0.70 to 0.95.

Security – 60 tasks (38 red‑team, 22 blue‑team) using real CVEs. The pipeline is clue → cause → reproduce → evidence. Scoring is deterministic; no LLM judge is used.

Benchmark leaderboard (average of three think‑mode runs)

Code : Claude Opus 4.8 – 74.43 (cbc), 77.90* (cc)

Web : Claude Opus 4.8 – 68.14 (cbc), 69.86 (cc)

Office : Claude Opus 4.8 – 82.37 (cbc), GPT‑5.5 – 86.05 (cc)

Security : GLM‑5.2 – 76.32 (cbc), 80.86 (cc)

No single model dominates all tracks. Changing the Harness (the runtime wrapper) also shifts scores; for example, HY‑3 gains 3.82 points on Code under cbc.

Running the benchmark

uv sync
cp .env.example .env
./scripts/dataset/fetch-dataset.sh code
cp configs/models/_template.model.yaml configs/models/<provider>/<slug>.yaml
cp configs/jobs/_template.job.yaml configs/jobs/<slug>.yaml
uv run ./scripts/run.sh --job <slug> --dry-run
uv run ./scripts/run.sh --job <slug>

The job config must specify at least three fields:

model: provider/slug
harness: codebuddy-code/version
dataset: datasets/wb-bench-code-v1.0/tasks

Results are stored in results/. Scores indicate overall performance; detailed failure analysis requires inspecting patches, logs, token usage, and test outcomes.

Key observations

Fixing a bug proves only a local action. True completion requires locating the correct object, understanding contextual constraints, leaving usable artifacts, keeping state consistent, and providing auditable evidence for the next person.

Four tracks share a common task packaging and execution protocol, but each uses its own scoring criteria based on the nature of the deliverable.

Benchmark conditions that affect scores include model version, Harness version, dataset version, tool permissions, and instruction protocol. Recording all these factors is essential for reproducible comparison.

Cost analysis shows token usage varies by model: DeepSeek‑V4‑Flash averages ~28.6k tokens per task (lower score), while GPT‑5.5 averages ~8.7k tokens (higher score).