The Illusion of Readiness in Health AI

Large language models have demonstrated remarkable performance in a wide range of medical benchmarks. Yet underneath the seemingly promising results lie salient growth areas, especially in cutting-edge frontiers such as multimodal reasoning. In this paper, we introduce a series of adversarial stress tests to systematically assess the robustness of flagship models and medical benchmarks. Our study reveals prevalent brittleness in the presence of simple adversarial transformations: leading systems can guess the right answer even with key inputs removed, yet may get confused by the slightest prompt alterations, while fabricating convincing yet flawed reasoning traces. Using clinician-guided rubrics, we demonstrate that popular medical benchmarks vary widely in what they truly measure. Our study reveals significant competency gaps of frontier AI in attaining real-world readiness for health applications. If we want AI to earn trust in healthcare, we must demand more than leaderboard wins and must hold AI systems accountable to ensure robustness, sound reasoning, and alignment with real medical demands.

💡 Research Summary

The paper “The Illusion of Readiness in Health AI” presents a systematic investigation of the robustness of state‑of‑the‑art large language models (LLMs) when applied to multimodal medical diagnostic tasks. While recent LLMs such as GPT‑5, GPT‑4o, Gemini‑2.5 Pro, and various OpenAI models achieve impressive headline scores on popular benchmarks like NEJM and JAMA, the authors argue that these numbers mask serious deficiencies in visual reasoning, prompt sensitivity, and logical fidelity.

To expose these hidden weaknesses, the authors design six adversarial stress tests that progressively perturb the input data or the task format: (1) removal of the diagnostic image, (2) a visual‑required subset where images are essential, (3) random reordering of answer choices, (4) replacement of distractor options with “Unknown.” or unrelated choices, (5) substitution of the original image with a clinically plausible but incorrect image, and (6) audit of the model‑generated reasoning trace. Each test is evaluated on both the full image‑plus‑text condition and a text‑only condition, allowing the authors to separate true multimodal grounding from shortcut reliance.

Key findings include:

• Modality Sensitivity (Test 1). Removing images leads to substantial accuracy drops on NEJM (‑9 to ‑27 percentage points) and modest drops on JAMA. Some models retain surprisingly high performance without visual input, indicating reliance on textual cues or memorized image‑text associations.

• Visual‑Necessity Subset (Test 2). On a curated 175‑question NEJM‑VS set that clinicians deem image‑dependent, models still achieve 30‑40 % accuracy when the image is omitted—far above the 20 % random baseline. GPT‑4o is an outlier, abstaining on 97 % of such cases, which the authors interpret as a more appropriate uncertainty handling behavior.

• Format Sensitivity (Test 3). Randomizing answer order harms performance in the text‑only condition (‑5 to ‑6 pp), revealing that models exploit positional biases. When images are present, visual grounding mitigates this effect, sometimes even improving scores.

• Distractor Manipulation (Test 4). Replacing a distractor with “Unknown.” causes models to treat this token as an easy elimination cue rather than a fallback, inflating accuracy in the text‑only setting. Systematically swapping 1‑4 distractors with unrelated options gradually drives accuracy toward chance, confirming that many models depend on superficial distributional patterns when visual evidence is unavailable.

• Visual Substitution (Test 5). Swapping the correct diagnostic image with a plausible but wrong image leads to dramatic performance declines (‑30 to ‑35 pp) for all models except GPT‑4o, which remains relatively stable. This demonstrates that most LLMs do not truly re‑evaluate their answer based on changed visual evidence, but rather rely on static image‑answer pairings learned during pre‑training.

• Fabricated Reasoning (Test 6). Even when a model guesses the correct answer, the accompanying explanation often contains factual errors, hallucinated visual descriptions, or reasoning that does not follow from the provided image. This “fabricated reasoning” undermines trustworthiness and raises safety concerns for clinical deployment.

Beyond the quantitative results, the authors introduce clinician‑guided rubrics that dissect each test item into sub‑skills such as image perception, clinical reasoning, and uncertainty management. Applying these rubrics reveals that existing multimodal medical benchmarks are heterogeneous: some primarily assess textual knowledge, others rely heavily on visual cues, yet they are frequently treated interchangeably in the literature.

The paper concludes that current leaderboard victories do not guarantee real‑world readiness. Robustness to missing or corrupted inputs, resistance to prompt engineering tricks, and the ability to produce faithful, clinically sound explanations are essential prerequisites for trustworthy health AI. The authors advocate for a paradigm shift toward stress‑test‑driven evaluation, richer benchmark design, and stricter accountability standards before large language models are integrated into patient‑facing or decision‑support systems.