In medical annotation and crowdsourcing, it is common for multiple annotators to label each example. The resulting labels are not always correct, and the new arXiv paper (2610.19543) points out that treating these noisy labels as ground truth can bias calibration evaluation.
Calibration measures how well a model's predicted probabilities align with actual outcomes. If the reference labels themselves are unreliable, the evaluation becomes misleading, potentially hiding poor calibration or suggesting accuracy where none exists.
The abstract states that the paper aims to address this problem, but the provided text cuts off before describing the proposed approach. As a result, readers are left with a clear statement of the issue but no details on the solution or experimental results.