A recent Nature Medicine study made headlines with the finding that ChatGPT Health under-triages 51.6% of emergency cases, concluding that consumer-facing AI triage tools pose safety risks. But a new analysis posted on arXiv argues that this result may be an artifact of how the test was administered.
The critique focuses on the protocol: the original evaluation used an exam-style scaffold that forced the model to choose among A/B/C/D options. According to the arXiv authors, this format—rather than the model's underlying capability—may be what drives the measured failure rate. They argue that evaluation design deserves as much scrutiny as model performance when assessing consumer health AI.
Where the two studies agree is that triage accuracy matters; where they differ is on what the 51.6% figure actually represents. The new paper does not dispute the number itself, but questions whether it is a valid measure of real-world triage behavior.