Direct Preference Optimization (DPO) trains language models on pairwise human preferences using a logistic objective derived from the Bradley–Terry model. A new theoretical paper asks what happens when that model is wrong. The authors distinguish between three failure modes: the score function class is too limited, the data is too small, or the preference structure itself cannot be represented by Bradley–Terry at all.
The core contribution is a quantity called the triangle-curl of the preference log-odds. It vanishes exactly when the preferences admit a Bradley–Terry representation, giving a precise characterization of structural misspecification. The authors prove upper bounds on excess logistic risk that decompose into a score-class approximation term plus a triangle-curl term, and they show a matching lower bound proportional to triangle-curl under boundedness assumptions.
For ReLU-based score models, including clipped variants, the paper derives explicit approximation rates: increasing capacity can drive the score-approximation term to zero, but the triangle-curl term remains untouched. The authors also provide finite-sample bounds for clipped empirical risk minimizers that separate network approximation, structural misspecification, and estimation error. They connect these results to a surrogate metric called contextual preference accuracy (CPA), while cautioning that CPA is not a complete measure of generated-response quality.