A new preprint on arXiv examines how large language models handle medical evidence when multiple publications stem from the same clinical trial. The authors argue that publication count is not the same as study count, and that a model may know two papers share a lineage yet still treat the second paper as independent evidence during aggregation.
In their evaluation, the model performed well on a direct task of recognizing shared trial lineage. However, when asked to aggregate evidence across sources, it over-credited a second publication from the same trial, effectively double-counting the underlying study. This gap between explicit recognition and applied reasoning could distort automated medical verification systems.
The findings highlight a subtle failure mode: even when a model can state that two papers come from the same trial, it may not adjust its confidence accordingly. The authors suggest that future systems need to integrate lineage awareness into the aggregation step, not just into a separate classification task. The paper is available on arXiv under identifier 2610.23721.