A new arXiv study probes how much language actually contributes to pathology vision-language models (VLMs), which are often touted as making whole-slide images queryable and able to draft reports. The authors tested four common claims using controlled experiments that reuse released checkpoints and public data, varying one input at a time. Their goal was to separate genuine gains from artifacts of evaluation.

On the positive side, report text adds a small but real prognostic signal. Embedding a stage-redacted pathology report alongside a slide embedding raised the pooled C-index by 0.019 for progression-free interval and 0.016 for overall survival across 1,633 TCGA patients. Restoring the stage statement added nothing, suggesting the signal comes from other report content. However, prompts at inference time provided no measurable accuracy boost: zero-shot classification was statistically indistinguishable from a supervised linear probe for lung and renal cancer subtyping.

Question answering was barely grounded in the image. SlideChat answered 74.82% of SlideBench-VQA(TCGA) items correctly with the slide, but 74.41% without it, while its language backbone alone reached 43.52%. Paraphrasing left accuracy unchanged but flipped 8–23% of individual answers, indicating instability. The authors conclude that language adds real but narrow value, and they propose minimum controls—such as ablating the image and paraphrasing prompts—that future evaluations should adopt to expose these failure modes.