Chatbot Arena's leaderboard ranks language models using the Bradley-Terry model, a statistical framework that gives each model one strength score. That design implicitly assumes Luce's choice axiom, a standard condition in choice theory about how preferences scale across options.

The new arXiv paper (2610.19095) applies choice theory to predict where this assumption breaks down. The key case is near-copies: models that are almost identical. According to the abstract, such models are told apart more decisively than the axiom would predict.

The implication is that small gaps between similar models in the Arena rankings may not reflect real quality differences, but rather a failure of the underlying model under Luce's axiom. This is a single-source analysis, so no independent confirmation is available yet.