Medical vision-language models (VLMs) often appear to perform well on average, but this can conceal a problem known as hidden stratification. The models may systematically fail on subtle visual findings or overlapping clinical conditions, even while aggregate metrics look strong. The new arXiv paper, ViCoR, attributes this failure mode to modality imbalance — a tendency for the model to lean on the text modality rather than genuinely processing the visual input.
To address this, the authors propose estimating "visual necessity" through counterfactual residuals. The idea is to measure how much the model's prediction depends on the image by comparing it to a counterfactual scenario where the visual input is altered or removed. ViCoR then uses this signal to select training data that forces the model to attend to the image, rather than defaulting to text-only shortcuts.
The approach is presented as a data-selection strategy rather than a new architecture, which could make it easier to integrate into existing medical VLM pipelines. By focusing on cases where visual information is truly necessary, ViCoR aims to reduce hidden stratification and improve performance on the difficult, visually demanding cases that average metrics often miss. The abstract does not provide quantitative results, so the claimed benefits are best understood as a proposed mechanism for future validation.