Object detection benchmarks treat annotated test sets as ground truth, but annotations are often noisy or ambiguous. Annotators may disagree on whether an object is present, where its bounding box sits, or what class it belongs to. Standard agreement metrics typically binarize box overlap—turning a continuous overlap score into a match/no-match decision—before computing a summary statistic. That loses information about the degree and structure of disagreement.

The authors propose an Optimal Transport (OT)-based metric that instead assigns graded costs to matched and unmatched labeled boxes. This keeps the comparison continuous and lets the metric reflect how far apart annotations are, not just whether they overlap beyond a threshold. Through theoretical analysis and experiments on real and synthetically perturbed annotations, they show the OT metric captures graded changes in agreement and can distinguish annotation pairs that thresholded metrics score identically.

The work is accepted to ACCV 2026. The authors argue the approach is especially useful for domains with many small or ambiguous objects, where thresholded overlap is a blunt instrument. Finer-grained agreement estimates could support more reliable validation when labels are noisy or conflicting.