Aligning large language models with human preferences typically relies on pairwise feedback, where annotators choose between two responses. A new arXiv paper argues that ranked feedback—ordering multiple responses within a single annotation—can capture more comprehensive information, potentially reducing annotation effort while improving alignment.
The paper, titled "Divergence based Ranked Preference Alignment for LLMs," proposes a method that uses divergence measures to align models with ranked preferences. The provided abstract is truncated, so the specific mathematical formulation and experimental results are not available in the source.
If successful, this approach could offer a practical alternative to pairwise methods by leveraging richer feedback signals from each annotation, though the full details and validation remain to be seen in the complete paper.