Multiple-choice tests are a common way to evaluate language models, but they have a known weakness: reordering the answer options can change which answer the model gives. Researchers have studied this problem from two angles—selection bias, where a model prefers certain option letters, and order sensitivity, where answers shift across different orderings. A new paper on arXiv connects these two lines of work.

The paper, titled "One Bias Score, Two Factors," proposes a unified way to think about the problem. Instead of treating selection bias and order sensitivity as separate issues, the authors suggest a single bias score that can be decomposed into two factors. This makes it possible to measure how much of the bias comes from each factor and to see how they interact.

The abstract also indicates that the paper looks at how mitigation strategies affect option bias. By separating the two factors, the authors can examine whether a mitigation reduces one factor, the other, or both. The implication is that evaluation designers need to be aware of both factors, since a mitigation that addresses one may leave the other untouched.