Why LLMs Fail at Real-World Medical Triage Despite Passing Exams

TL;DR: A study of 2,400 MIMIC cases reveals that LLMs perform significantly worse than physicians in diagnosing abdominal pathologies due to an inability to handle incomplete data. This highlights a critical safety gap for autonomous clinical triage where human oversight remains essential.

Key facts

• LLMs pass medical licensing exams but lack safety behaviors for autonomous clinical triage, especially with incomplete patient data.
• Optimized for statistical text generation, LLMs fail to broaden differentials or escalate care when facing rare, life-threatening conditions.
• A study of 2,400 real MIMIC database cases showed current LLMs performed significantly worse than physicians on four abdominal pathologies.
• LLMs in the MIMIC study failed to follow diagnostic guidelines, misinterpreted lab results, and struggled with workflow integration.
• Model performance was highly sensitive to both the amount and order of information provided during clinical evaluation.
• Existing evaluations often use complete simulations that mask critical failure modes like positive bias and inability to handle ambiguity.
• Researchers propose multi-agent systems with specialized modules for vital signs and lab analysis to improve interpretability in intensive care.

The Promise and Peril of AI in Clinical Triage

Large language models (LLMs) have demonstrated the ability to pass medical licensing examinations and perform diagnostic reasoning on carefully selected, well-documented patient cases [1]. These achievements have fueled growing interest in deploying LLMs for clinical decision support—ranging from symptom assessment to treatment guidance and administrative documentation. However, a recent perspective warns that these successes do not translate into safe autonomous use in real-world settings, particularly during the critical phase of triaging undifferentiated patients [1].

The Core Deficit: Missing What Matters Most

The primary issue lies in how LLMs are trained and evaluated. Rather than being optimized for clinical safety, they are designed to generate text that is statistically likely based on their training data. This creates a fundamental mismatch in high-stakes environments where the cost of missing a rare but life-threatening condition far outweighs the risk of a false alarm [1]. In such cases, clinicians are trained to broaden their diagnostic thinking, actively seek out red flags, and escalate care when uncertainty is present. LLMs, however, often fail to exhibit these behaviors. They do not reliably broaden differentials, probe for missing information, lower escalation thresholds, or defer judgment when evidence is insufficient.

Real-World Performance Falls Short

Empirical testing reveals the gap between lab performance and real-world application. A study using 2,400 real patient cases from the Medical Information Mart for Intensive Care (MIMIC) database evaluated state-of-the-art LLMs on four common abdominal pathologies [4]. The results showed that current models performed significantly worse than physicians in making accurate diagnoses across all conditions. Moreover, the models failed to follow established diagnostic and treatment guidelines, misinterpreted laboratory results, and struggled to integrate into clinical workflows due to sensitivity to both the amount and order of information provided [4].

Flawed Evaluation Methods Mask Critical Failures

Many existing evaluations rely on complete, curated simulations that do not reflect the ambiguity and incompleteness of real patient histories. In these controlled settings, LLMs may appear competent—displaying assistant-like behaviors and positive biases—that give a misleading impression of reliability [1]. This creates a false sense of security, as the models’ inability to handle missing or contradictory data remains hidden. The lack of stress-testing under uncertainty means critical failure modes go undetected until deployment in live clinical environments.

Toward Safer Systems: Multi-Agent and Ethical Governance Approaches

Researchers are exploring alternative architectures to improve safety and interpretability. One approach involves multi-agent systems, where specialized agents handle different aspects of patient data—such as interpreting vital signs, analyzing lab results, or reasoning about context—before synthesizing a final recommendation [3]. This modular design aims to enhance transparency and trust, particularly in intensive care settings. Ethical AI governance frameworks are also being developed to ensure accountability and alignment with clinical safety standards.

The consensus remains clear: LLMs are not yet ready for autonomous decision-making in clinical care. Their performance on exams and curated cases does not guarantee safe behavior in real-world triage, where incomplete information is the norm rather than the exception. Until models can reliably prioritize ‘must-not-miss’ diagnoses and adapt safely under uncertainty, human oversight remains essential.

Key Facts

• LLMs have passed medical licensing examinations and can rival physicians at diagnostic reasoning in curated cases, but evidence of safety for autonomous triage does not yet exist [1,2].
• The core deficit of LLMs in clinical care is a lack of fidelity in clinical evaluation, specifically the inability to act safely when the correct answer is an improbable ‘must-not-miss’ diagnosis [1,2].
• LLMs optimized for probable text generation fail to show safe triage behaviors such as broadening differentials, seeking missing red flags, lowering escalation thresholds, or deferring judgment under incomplete histories [1,2].
• Current LLM evaluations often use complete, well-curated simulations that do not test reasoning under missing information, potentially masking failure modes like assistant-like behaviors and positive bias [1,2].
• A study using a simulated clinical setting based on the MIMIC database with 2,400 real patient cases found that current LLMs performed significantly worse than physicians in diagnosing four common abdominal pathologies [4]
• In the same study, LLMs failed to follow diagnostic or treatment guidelines and could not interpret laboratory results accurately [4]
• LLMs demonstrated poor integration into clinical workflows because they are sensitive to both the quantity and order of information provided [4]
• Multi-agent systems utilizing modular agents for lab analysis, vital signs interpretation, and contextual reasoning have been proposed to improve interpretability and trust in intensive care settings [3]

Sources

1. Reasoning in Real World Clinical Care: Why Large Language Models Are Not Yet Safe for Autonomous Clinical Decision Support (arxiv.org) — 2026-07-29
2. Evaluation and mitigation of the limitations of large language models in clinical decision-making (pmc.ncbi.nlm.nih.gov) — 2024-07-04
3. Reinforcing Clinical Decision Support through Multi-Agent Systems and Ethical AI Governance (arxiv.org) — 2025-02-19