Most benchmarks for large language models in medicine are built from licensing examinations and theoretical questions. The few that draw on real clinical data do so only in part, and rarely require a model to read a patient's own records, according to a new paper on arXiv.
The paper introduces ClinEdge, an expert-graded benchmark of real patient edge cases. It is designed to test frontier multimodal models on scenarios that reflect actual clinical complexity. The title states that these models fail on such cases, pointing to a gap between theoretical knowledge and real-world reasoning.
The significance of ClinEdge is that it offers a more demanding evaluation than traditional exam-style tests. By focusing on edge cases from real patients, it may better expose limitations that licensing-exam benchmarks miss, suggesting that current assessments could overstate model capability in clinical settings.