FDE-Gym Benchmark Exposes AI Delivery Gap: Top Models Solve Only 14 of 95 Tasks
Tsinghua, Microsoft, and collaborators introduce FDE-Gym, a reproducible benchmark for AI delivery agents across 20 customer scenarios and 95 tasks, revealing that even Claude Opus 5.5 solves only 14 tasks at the most realistic level, with the core bottleneck being discovery of implicit constraints and translation of facts into deployed systems.
Introduction: Code Generation ≠ Delivery
The article opens by highlighting a critical gap: AI can write code, but delivering that code into a real production environment — where unstated constraints must be discovered and business metrics must improve — remains unsolved. FDE-Gym, released by researchers from Tsinghua University, Shanghai Jiao Tong University, Harvard, Stanford, Microsoft, and Meta, is the first benchmark that turns this complex delivery process into an executable, reproducible Agent environment.
Benchmark Scope: 20 Scenarios, 95 Tasks, 18 Industries
FDE-Gym covers 20 customer scenarios and 95 end-to-end tasks spanning 18 industries. Each scenario includes a customer, a task initiator, and a production system to be modified; the deliverable can be an Agent, a workflow, or a service. To prevent template memorization, the team designed 2–7 variants per original task , altering facts such as expiration rules, key stakeholders, and requirement drift. Agents must re-verify each variant in the same environment.
Over 130 stakeholders are simulated via scripts with fixed knowledge boundaries, misunderstandings, patience levels, disclosure conditions, and contact limits. This preserves the communication game while ensuring reproducible interactions.
End-to-End Delivery Loop: Workspace to Production
Agents start with four artifact types: task description, a copy of the production system, documentation and historical data, and an employee directory. The runtime provides only standard Python and SQLite — no arbitrary package installs or public internet access. Network requests from the delivered system can only go to the customer's LLM gateway and are metered by usage.
After delivery, the system/ directory is deployed verbatim; the production system continues calling its original entry points. Writing an isolated script does not count — new logic must be wired back into the real call chain. The system then runs on unseen production data. Functional tests passing is only an intermediate state; the final judge is whether customer KPIs improve.
Four-Level Instruction Design: Controlled Variable Experiment
The paper uses a clean controlled-variable design: workspace, employees, and evaluator stay fixed; only the information density in the task description changes.
L0 : Real problem, rules, constraints, and scoring method given — tests engineering implementation.
L1 : Rules and constraints hidden — forces Agent to discover them actively.
L2 : Only a directionally correct vague requirement provided.
L3 : Original initiator request restored, which may even point to the wrong problem — closest to real delivery.
The drop from L0 to L1 mainly reflects constraint discovery ability ; further decline after L1 reflects problem clarification ability . This disentangles requirements understanding from coding skill.
Scoring: Business Improvement, Not Functional Completion
FDE-Gym measures delivery value with a relative uplift metric: U = (Delivered System KPI − Original System KPI) / (Reference Solution KPI − Original System KPI) . U = 0 means no improvement; U = 1 means matching the expert reference; U < 0 means making things worse.
The final score applies two adjustments: (1) predefined incidents (e.g., privilege escalation) zero out positive gains; (2) communication cost C (questions, trial runs, waiting) is subtracted: S = Incident-adjusted U − C . A task counts as solved only if S ≥ 0.8 with no incidents.
The evaluator underwent stress tests: reference solutions ≥ 0.95, literal execution ≤ 0.2, shortcut strategies ≤ 0.8, and 270+ bypass/speculative strategies verified anti-gaming robustness.
Experimental Setup: 13 Model Configurations, Minimal Framework
Experiments covered 13 model configurations from 7 providers. All models used the same minimal Agent framework with only five tool categories (command execution, file read/write/edit, etc.), no model-specific customization. Each combination ran 3 times under unified caps: 200 turns, 150 minutes, $8 cost.
Key Results: Best Model Solves Only 14/95 Tasks at L3
At the most realistic L3 level:
Claude Opus 5.5 leads with score 0.56, solving only 14 of 95 tasks .
Claude Sonnet 5.5 scores 0.49, solving 15 tasks.
Even an oracle that picks the best model per task reaches only 0.70 overall (30/95 solved).
Chinese models Deepseek-V4.1-Flash and GLM-5.3-Flash rank near the top.
When full information is given (L0), leading models score 0.73–0.82, showing substantial engineering capability — but much business value remains unrealized even then.
Constraint Discovery Is the Steepest Cliff
Leading models drop 0.28 points on average from L0 to L1 (rules/constraints hidden), but only 0.04 further from L1 to L3 . The sharpest fall occurs when implicit constraints must be discovered autonomously, not merely when requirements become vague.
Fact Retrieval ≠ Business Improvement
Top models already acquire 72%–85% of key facts , yet business uplift reaches only 0.60–0.70 of the reference solution. Opus 5.5 retrieved all facts on 23 tasks, yet only 10 achieved ≥ 0.8 business improvement.
The bottleneck is fact-to-system translation : how to encode rules, handle exceptions, maintain backward compatibility, and validate before go-live. Repeated questioning also burns 0.14–0.22 score points, so communication strategy must balance information gain against customer cost.
Three System Capabilities for Next-Generation Delivery Agents
Constraint Tracking : Link evidence from docs, conversations, and code to specific rules, annotating source, timeliness, and unknowns — preventing fact loss during implementation.
Cost-Aware Decision Making : Before asking, check existing evidence; prioritize contacts who can change the solution; fold trial runs and waiting into a unified budget.
Deployment Protection : Connect business KPIs via holdout data, regression tests, and canary validation; retain a no-change rollback option.
This capability boundary is practical: Claude Haiku 4.5 scored below the original system on 79 tasks ; Mistral Small 4 had 51 tasks below baseline even before communication cost. Delivery agents must learn when not to act.
Limitations and Future Directions
FDE-Gym currently uses scripted employees and a single minimal framework; human expert baselines are still pending. The most valuable next step is tracking whether stronger delivery frameworks can close the engineering translation gap. The benchmark already points a clear direction: the true last mile is closing the loop among requirements, constraints, code, and business outcomes on a single chain.
Resources
https://arxiv.org/abs/2610.22350
https://fde-gym.com
https://github.com/deeplethe/fde-gymSigned-in readers can open the original source through BestHub's protected redirect.
This article has been distilled and summarized from source material, then republished for learning and reference. If you believe it infringes your rights, please contactand we will review it promptly.
How this landed with the community
Was this worth your time?
0 Comments
Thoughtful readers leave field notes, pushback, and hard-won operational detail here.