A new arXiv preprint highlights a mismatch in how medical vision-language models (VLMs) are evaluated. These models are typically tested on clean, high-quality inputs, even though clinical settings may involve progressive loss of evidence in the images they must interpret.
The paper notes that existing robustness studies have already shown that performance can degrade as evidence becomes less reliable. However, standard evaluation practices still overlook this progressive degradation, leaving a gap between benchmark results and real-world reliability.
The title's reference to a "behavioral mirage" suggests that models may appear to function well until a critical threshold is crossed, though the abstract does not elaborate on this point. The authors call for evaluation methods that account for progressive evidence loss in medical imaging.