Tencent’s WorkBuddy Unveils Its Internal Benchmark in a New Paper

1. WorkBuddy Advantages

WorkBuddy Bench’s advantage is distribution‑informed task alignment with full auditability.

Four subsets each have independent scoring instruments; scores are not comparable across tracks, and the suite deliberately omits an overall average.

2. Core Methodology: Preventing Prompt Pollution from Real‑World Data

The paper treats searchable prompts as the primary contamination path and blocks them during task generation.

Open‑source checklist (Table 2) lists everything needed to run, score, and audit a task.

Task source : each task anchors to a real upstream artifact (e.g., a commit/PR or a CVE) and matches the internal distribution of intent categories and request patterns; raw user prompts are never used.

Rewritten protocol : after reverse‑engineering, prompts are shortened, conversational, and deliberately underspecified; the Code track also uses five roles (developer, algorithm engineer, product manager, QA, ops).

Deliberate underspecification : only intent and constraints are provided, no target files or interface definitions, forcing the agent to retrieve missing context from the workspace, testing disambiguation and grounding.

Post‑run isolation : scoring assets are invisible to the agent during execution and are injected only after the run; “Hidden tests” are released together with the open‑source benchmark.

3. Four Track Decomposition

3.1 Code

The Code subset places the agent in a checkout of a baseline commit of an open‑source repository, requiring it to locate, modify, and pass hidden tests. Compared with SWE‑bench, it adds five roles and 18 sub‑categories; only 10 of 80 tasks involve bug fixing.

Task admission requires a baseline reward ≤ 0.3 and an oracle reward = 1.0 after applying a gold patch. Early evaluations exposed two zero‑score failure modes: endless loops editing test files until timeout, and getting lost in large repositories, modifying unrelated files. The difficulty stems from navigation and locating code rather than synthesis.

3.2 Web

The Web subset enforces an “artifact‑not‑chat” contract: the agent must produce a runnable artifact at a declared output path (e.g., an HTML entry). If the path is empty, the task scores zero. Seventy tasks span generation, modification, analysis, and quality‑check of front‑end work.

3.3 Office

The Office subset evaluates the final workspace state of mixed‑format files (spreadsheets, documents, PDFs, JSON, Markdown, file trees). Scoring focuses on the end state; a plausible textual summary that does not update the referenced workbook still loses points.

Scoring uses a dual channel: deterministic rule checks (file, schema, numeric, cross‑file relations, state changes, side‑effect boundaries) and an LLM judge that assesses a fixed‑evidence rubric based on post‑run artifacts. Each task assigns its own rule weight w_i ∈ [0.70, 0.95].

3.4 Security

The Security subset covers red‑team and blue‑team scenarios: vulnerability discovery and exploitation (32 red), malware analysis (14 blue), security operations (8 blue), and agent security (6 red). Tasks reproduce real historical CVEs in projects such as binutils, curl, nginx, vim, jq, fluent‑bit.

Each task follows a two‑step structure: find‑vuln to locate the vulnerability path, then poc‑verify to submit a PoC that triggers an ASAN crash in a sandbox.

4. Leaderboard Results

Overall scores (0–100, think mode, average of three runs) are shown in Table 6. No single model dominates all tracks; a change in Harness configuration reshuffles rankings.

In the Code track, GPT‑5.5 leads under the cbc metric (72.90 vs 71.54) but is overtaken under cc (76.63 vs 77.06).

Security rankings shift dramatically: average absolute displacement is 8.6 points; GPT‑5.5 jumps from sixth under cbc to second under cc, while MiniMax‑M3 falls from second to fifth.

Office scores are stable: five of seven double‑run models move less than 2 points (median 0.86).

Engineering tweaks can add ~4 points: enabling cross‑round reasoning pass‑back improves HY‑3’s Code scores by +3.82 (cbc) and +1.92 (cc).

Refusals are recorded: Opus 4.8 exhibits 13 task‑level refusals on security tasks under Claude Code, while GPT‑5.5 has 2 refusals under cbc.

The hardest categories in the Code subset are bug_fix and api_contract (average 0.47), reflecting real‑world regression fixes and strict contract adherence; the easiest are feature_pipeline (0.94) and testing (0.88). Performance gaps are driven by business‑semantic understanding rather than code executability.

Token usage does not correlate with scores: DeepSeek‑V4‑Flash outputs ~3.3× more tokens than GPT‑5.5 under cc (28.6k vs 8.7k) yet scores 14.74 points lower. GLM‑5.2’s cc top score (77.06) consumes 22.0k output tokens, while GPT‑5.5 achieves 76.63 with only 8.7k. GPT‑5.5 is the most “token‑efficient high‑scorer” across all four tracks.

5. Positioning Against Existing Work

Table 8 compares the Web‑direction benchmark matrix with prior efforts, highlighting coverage breadth and framework differences rather than new task types. The Security subset uniquely offers full red‑blue coverage with deterministic scoring; Office treats mixed‑format hand‑off as a complete delivery; Code expands to five roles and 18 categories beyond simple issue fixing.

https://arxiv.org/pdf/2607.20911
https://workbuddybench.com/