Pairwise relevance judgments are a common way to evaluate reranking systems, but they complicate nDCG because they affect both the loss numerator and the ideal-ranking denominator. This paper, posted on arXiv, addresses that complication directly.
The authors characterize the exact worst-case loss of replacing a ranking with a fixed proposal. According to the abstract, binary gains attain this loss, meaning the worst case is realized by simple relevance labels rather than graded ones.
The announcement's abstract is cut off mid-sentence, so the full derivation and any broader claims are not visible from the source. Still, the result provides an exact risk certificate for pairwise-judged reranking, which could support tighter performance guarantees.