Microsoft's DxO Achieves 83% Accuracy in Sequential Clinical Diagnosis Benchmark

Clinical diagnosis is a sequential process of gathering information rather than a one-shot classification. Traditional medical AI benchmarks provide complete records upfront, making it difficult to evaluate whether a system actively asks the right questions, avoids unnecessary tests, or continues investigating when evidence is insufficient.

SDBench: A Sequential Diagnosis Benchmark

To address this limitation, Microsoft Research developed the Sequential Diagnosis Benchmark (SDBench), published in Science (October 22, 2026) as "Sequential diagnosis with language models" ( https://www.science.org/doi/10.1126/science.aeh7930). SDBench redesigns diagnosis as a multi-stage information-gathering and decision-making process. SDBench uses two high-quality case sources:

First set (304 cases): Complex, difficult cases from New England Journal of Medicine (NEJM) Clinical Pathological Conferences, emphasizing high diagnostic difficulty and teaching value.

Second set (325 cases): Real-world emergency department admission records from the MIMIC-IV-ED database, reflecting routine emergency scenarios.

Each case is converted into a brief description without the final diagnosis. A Gatekeeper agent releases requested information (history, physical exam, lab, imaging) on demand, preventing the diagnostician from seeing the full record prematurely. For tests not documented in the original case, the system generates plausible simulated results consistent with the case context to avoid leaking clues through missing data.

DxO: Diagnostic Orchestrator

DxO is a model-agnostic orchestrator that simulates a team of complementary clinical expert roles. In each step it maintains and updates a differential diagnosis, selects tests that help distinguish candidate diseases, challenges the current leading hypothesis, and evaluates test cost versus diagnostic value. The system updates its judgment after receiving new information, then decides whether to ask more questions, order tests, or submit a final diagnosis. It tracks testing expenditure, enabling explicit assessment of the accuracy-cost trade-off.

The orchestration aims to mitigate premature closure — where an early preference for a disease causes subsequent information to be interpreted as supporting that hypothesis — by continuously proposing counter-evidence and re-comparing candidates. Cost information also influences test selection when diagnostic value is similar.

Diagnostic Accuracy Results

Using GPT-5 as the base model, DxO achieved:

NEJM complex cases (304): 83.2% accuracy. Participating physicians averaged 22.3% without external references and 36.1% with search/reference access. The best individual physician reached 51.8%.

Emergency department cases (325): 80.3% accuracy. Physicians averaged 46.8%, with the top performer at 62.5%.

These results show DxO can leverage continuously collected information to achieve high diagnostic accuracy on both case sets. However, the comparison conditions must be considered: NEJM cases are selected for teaching difficulty and do not represent common outpatient distributions; MIMIC-IV-ED cases are specific to admitted ED patients. Physicians worked independently, not as specialty consultation teams. Therefore, accuracy numbers cannot be directly extrapolated to real-world clinical performance.

Accuracy-Cost Trade-offs

DxO's testing costs depend on budget constraints:

No budget limit: NEJM cases averaged $5,431 per case (physicians $4,725); ED cases averaged $4,979 (physicians $3,400).

With budget constraints: System reduces tests, trading some accuracy for lower cost. On NEJM cases, one configuration achieved 80.9% accuracy at $2,241; another 79.6% at $1,303. On ED cases, configurations yielded 73.2% at $2,634 and 69.5% at $2,037.

The research does not claim DxO simultaneously achieves highest accuracy and lowest cost in all settings; rather, it demonstrates adjustable diagnostic strategies under different budget constraints.

Limitations and Future Work

SDBench currently relies on text-based cases and does not fully cover multimodal clinical information, nor does it replicate real-world challenges like incomplete patient histories, ambiguous descriptions, or delayed test results. The study evaluates diagnostic-stage performance only and does not prove improvement in treatment planning, patient safety, or health outcomes. High accuracy on retrospective cases does not equate to independent clinical decision-making capability.

Microsoft plans a prospective study with Beth Israel Deaconess Medical Center where patients interact directly with the system, comparing its performance against independent physician assessments. In multiple test environments the system has been more accurate than human responders at lower total cost, showing promise as a physician-assist tool.