Generative AI in clinical diagnosis now has a statistical baseline, and it is more modest than the headlines suggest. The largest synthesis to date puts pooled diagnostic accuracy at roughly half — useful context before the benchmark wars get louder.
Where does medical AI stand today? The 83-paper answer
Generative AI is benchmark-competitive with average physicians but not with experts, and its measured diagnostic accuracy sits near 52%. The anchor is a systematic review and meta-analysis by Takita and colleagues in Nature's npj Digital Medicine, published 22 March 2025, which screened 18,371 records and synthesized 83 studies spanning June 2018 to June 2024 . It is the largest meta-analysis on the question so far, and its read-out frames expectations for everything that follows.
The pooled diagnostic accuracy across all configurations tested was 52.1% (95% CI 47.0–57.1%) . That number is an average over many models, case types, and study designs — not a single deployable system's score. Treat it as a center of gravity, not a product spec.
Coverage skewed heavily toward two OpenAI models, which matters when reading the aggregate:
| Model configuration | Articles evaluating it |
|---|---|
| GPT-4 | 54 |
| GPT-3.5 | 40 |
| GPT-4V, PaLM2, Llama 2, Claude 3 variants, Gemini | Also tested (smaller samples) |
GPT-4 was the most-evaluated configuration at 54 articles, with GPT-3.5 second at 40 . So the 52.1% figure mostly describes 2023-era OpenAI behavior, with newer entrants thinly represented.
The authors are explicit about what the pool is not: a neutral sample of real deployments. They flag high risk of bias and probable publication bias across the literature .
"Generative AI exhibits moderate diagnostic performance, comparable to non-expert physicians but below expert-level reliability," — Takita et al., npj Digital Medicine (source: Nature, 2025-03).
AI versus non-experts, versus attendings: the hierarchy
That single sentence hides two findings that sound contradictory but are not. In the same meta-analysis, generative AI showed no statistically significant difference against physicians overall (p=0.10) or against non-expert physicians specifically (p=0.93) . The defensible read: it matches the average generalist on text-based diagnostic reasoning.
The gap appears once you split out experts. Against expert physicians, AI trailed by 15.8 percentage points (95% CI 4.4–27.1; p=0.007) . That is both statistically significant and clinically meaningful — a margin wide enough to matter at the bedside, not a rounding artifact.
So both headlines describe one result:
- "Matches non-experts" — true at the generalist tier, where the p-values find no separation.
- "Trails specialists by 15.8pp" — equally true one tier up, where the gap is real.
- Quote either alone and you mislead; the honest claim carries both.
The deeper caveat is the comparator itself. "Physician" in these 83 studies pools residents, general practitioners, and subspecialty attendings into one label . A model "beating physicians" in one paper may face first-year residents; in another, board-certified subspecialists. The comparisons are rarely apples-to-apples, which is why a flat "AI beats doctors" headline collapses a hierarchy the data is careful to keep distinct.
For developers shipping clinical copilots, the practical takeaway is to ask which physician tier a benchmark used before trusting any parity claim — the answer changes what the number means.
Temporal instability: answer-flip frequency, calibration erosion, and staleness causes
A benchmark score is a snapshot, and the snapshot decays. When radiology in-training exam questions were re-administered to GPT-4 months later, the model picked a different answer 25.5% of the time . That instability — documented by Guan, Bates and Zhou in their review "Keeping Medical AI Healthy and Trustworthy" (IEEE Transactions on Biomedical Engineering, 10 Dec 2025) — is the variable a single benchmark win conveniently ignores.
The drift is not limited to LLMs answering test questions. The same review tracks classical clinical models eroding in production: in-hospital mortality prediction degraded measurably across 2.5 years of prospective deployment, and acute kidney injury (AKI) calibration slipped across a nine-year tracking window . Performance is a moving line, not a fixed point.
The causes split cleanly into two buckets:
- Data-level shift — covariate shift (the input distribution changes), label/prevalence shift (disease base rates move), and concept shift (the relationship between inputs and outcomes changes).
- Model-level decay — calibration drift, catastrophic forgetting, and training-cutoff staleness, where the model simply has not seen guidelines published after its knowledge cutoff.
"Model performance is not static after deployment; without continuous monitoring, degradation can go undetected until it reaches the patient," argue the authors, who propose a Detect–Diagnose–Correct framework to bound the decay (source: Guan, Bates & Zhou, IEEE TBME, 2025-12).
Two purpose-built benchmarks quantify the staleness problem. BioPulse-QA scores models on questions drawn from drug labels, trials and guidelines published after the model's training cutoff; clinical-trial extractive F1 fell as low as 0.36, meaning guideline currency degrades silently while headline accuracy looks fine . DriftMedQA ran seven models across 4,290 evolving-guideline scenarios and found they struggled to reject outdated recommendations; retrieval augmentation combined with preference tuning was the most effective intervention .
For anyone shipping a clinical copilot, the operational lesson is concrete: version your evaluations, re-test on a schedule, and treat a launch-day benchmark as perishable. The model that passed in March may quietly answer differently by autumn.
Orchestrated panels: Mira at 87.1%, MAI-DxO at ~80%, AMIE's OSCE lead
The headline scores driving the "AI beats doctors" narrative come from orchestrated, multi-step systems — not single prompts. Three stand out: Microsoft's MAI-DxO at roughly 80% on hard cases, Dresden/Heidelberg's Mira at 87.1% on emergency triage, and Google's Gemini-based AMIE matching or leading primary-care physicians in simulated visits. All three beat their human comparators on paper. All three are labelled research demonstrations by their own authors.
Microsoft's Sequential Diagnosis work turns 304 New England Journal of Medicine clinicopathological-conference cases into stepwise encounters, where the solver iteratively queries a gatekeeper model and is scored on accuracy and test cost . MAI-DxO, orchestrating a panel of model personas with OpenAI's o3, reached about 80% — roughly four times the ~20% average of 21 generalist physicians . The catch: those physicians worked without textbooks, colleagues, or web search.
Mira goes further into the clinical workflow. The German academic system operates over electronic-health-record data and can select from more than 85,000 clinical actions. Across 500-plus emergency-department cases, it hit 87.1% diagnostic accuracy on eight conditions — including pancreatic cancer, pneumonia, appendicitis, and pulmonary embolism — versus 78.1% for a six-physician specialist panel . Cases were presented through AI patient agents, which may have been more structured than real ED speech.
AMIE targets management, not just diagnosis. In a randomized, blinded virtual OSCE over 100 multi-visit cases against 21 GPs, it was non-inferior on management reasoning and stronger on guideline alignment, treatment precision, and medication questions drawn from US and UK national drug formularies .
| System | AI score | Physician comparator | Setting |
|---|---|---|---|
| MAI-DxO (o3) | ~80% | ~20% (21 generalists) | 304 NEJM CPC cases, sequential |
| Mira | 87.1% | 78.1% (6-physician panel) | 500+ ED cases, patient agents |
| AMIE | Non-inferior / led | 21 GPs | 100-case virtual OSCE |
None of these are released live-patient products, and none carry prospective, at-scale RCT validation. The pattern is consistent: orchestration and sequential querying lift scores well above one-shot prompting, but the comparison conditions flatter the machine.
Why the test format inflates or collapses apparent AI capability
The test design is itself a variable — exam-style forced-choice and free-text conversational setups can return opposite verdicts on the same underlying model. A Nature Medicine study reported in February 2026 ran ChatGPT Health across 60 scenarios in 21 specialties and found under-triage in 51.6% of emergency cases, plus inconsistent suicide-crisis guardrails — including one scenario where adding normal lab values suppressed the crisis banner entirely .
A March 2026 replication disputed that the failures were the model's. It argued the original used an exam-style forced multiple-choice setup, and that naturalistic, patient-style input changes the picture. In a partial replication across five frontier models, conversational framing improved triage accuracy by 6.4 percentage points . Specific conditions moved sharply:
- Diabetic ketoacidosis was triaged correctly in 100% of trials under naturalistic input .
- Asthma triage rose from 48% to 80% on the same models .
The replication authors frame the format itself as the confound rather than a neutral lens:
"A forced multiple-choice exam can manufacture safety failures that a naturalistic patient interaction does not reproduce — the evaluation harness, not just the model, is under test," — argument of the March 2026 replication (source: arXiv:2603.11413).
Evaluation difficulty governs headline numbers just as strongly. OpenAI's HealthBench — 5,000 multi-turn conversations scored against 48,562 physician-written criteria — produces a wide spread by model: GPT-3.5 Turbo 16%, GPT-4o 32%, and o3 60% . On the curated HealthBench Hard subset, the top score was still only 32% . Same models, different slice, near-doubled or halved results — which is why any single "AI scored X%" claim is only legible alongside the harness that produced it.
What real-patient prospective trials have found — and what they haven't
Prospective real-patient evidence is thin, and the strongest data point is narrow. In a single-arm feasibility study of 100 adults in ambulatory urgent care, Google's AMIE produced a differential that included the final chart-reviewed diagnosis in 90% of cases and ranked it top-three in 75%, with no real-time safety-supervisor stops triggered (source: AMIE feasibility study, 2026-03).
The catch sits in the same study: physicians still beat AMIE on management practicality and cost-effectiveness. The gap is on execution, not differential generation — the model lists the right possibilities but trails clinicians on turning them into practical, affordable plans.
No prospective, at-scale, live-patient randomized trial exists for MAI-DxO, Mira, or AMIE in their current configurations. The orchestrated-panel headlines came from simulated encounters, and Microsoft itself frames MAI-DxO and SDBench as research demonstrations (source: Microsoft, 2025-06).
"MAI-DxO and SDBench are research demonstrations, not released or live-patient-validated systems," — Microsoft AI, Sequential Diagnosis with Language Models (source: arXiv:2506.22405).
For engineers integrating clinical AI, the operational read is concrete. The 25.5% answer-flip rate on re-tested radiology questions means version-pinning alone will not stabilize a pipeline (source: Guan, Bates & Zhou, 2025-12). Their Detect–Diagnose–Correct framework prescribes:
- Continuous held-out monitoring against fresh, labeled cases — not a one-time benchmark pass.
- Automated distribution-shift detection across covariate, label, and concept drift.
- Scheduled re-evaluation intervals, since clinical ground-truth labels arrive late.
The takeaway: expert clinicians still outperform these systems on complex management, and durable real-world reliability is unproven. Treat any benchmark score as a perishable snapshot, and build doctor-plus-AI workflows around live, versioned, post-deployment evaluation rather than a single launch-day result.
Frequently asked questions
How accurate is AI at medical diagnosis compared to an average doctor in 2025–2026?
On text-based diagnostic reasoning, generative AI now matches non-expert physicians but trails specialists. A Nature npj Digital Medicine meta-analysis of 83 studies found a pooled diagnostic accuracy of 52.1% (95% CI 47.0–57.1%), with no statistically significant difference versus non-expert physicians (p=0.93) yet a 15.8-percentage-point deficit against expert physicians (p=0.007) . The defensible summary: matches generalists, not specialists.
Why do AI medical benchmark results degrade or expire over time?
Benchmark scores are perishable for three structural reasons:
- Model updates shift answer distributions. When radiology in-training exam questions were re-administered to GPT-4 months later, it picked a different answer 25.5% of the time .
- Training-cutoff staleness erodes performance on guideline-currency questions — BioPulse-QA saw clinical-trial extractive F1 fall as low as 0.36 .
- Clinical practice itself moves, producing covariate and concept drift in the underlying patient population.
What is MAI-DxO and how does it differ from a single-model evaluation?
MAI-DxO is Microsoft's Diagnostic Orchestrator: instead of querying one model once, it runs a panel of model "personas," routes stepwise diagnostic queries through a gatekeeper, and synthesizes a final answer. It was evaluated on 304 NEJM clinicopathological-conference cases via SDBench with cost tracking, reaching roughly 80% accuracy with OpenAI's o3 . It is a research demonstration, not a shipped or live-patient-validated product.
Is the 51.6% under-triage finding for ChatGPT Health reliable?
It is contested. The original Nature Medicine study reported under-triage in 51.6% of emergency scenarios, but a March 2026 replication argued the forced multiple-choice exam format drove most failures. Under naturalistic free-text interaction, triage accuracy rose by 6.4 percentage points and asthma triage climbed from 48% to 80% . The evaluation design — not solely model behavior — is the confounding variable.
What practical steps address temporal decay when integrating a clinical AI into a product?
Guan, Bates and Zhou propose a Detect–Diagnose–Correct framework: continuous monitoring on held-out clinical cases, automated distribution-shift detection, and scheduled re-evaluation intervals, since clinical ground-truth labels often arrive late . For guideline-currency queries specifically, DriftMedQA found retrieval augmentation combined with preference tuning gave the strongest results at rejecting outdated recommendations .
