Pairwise preference feedback is a common supervision signal for reinforcement learning in video generation models, but the way it is aggregated can be flawed. The authors of a new arXiv preprint define a preference graph for each GRPO group, with nodes as sampled rollouts and edges as pairwise comparisons. Auditing 6,976 such graphs, they found that cyclic preferences—where A beats B, B beats C, and C beats A—are prevalent and persist throughout training.
Naive aggregation based on average win rates treats each rollout independently and discards the underlying comparison structure. The authors argue this can produce unreliable advantage estimates for certain rollouts and hurt optimization. To address this, they propose the Reliability-Anchored Preference Walk, a graph-aware method that combines global win-rate signals with reliable relational evidence from cyclic graphs.
The method requires no additional evaluator queries beyond the comparisons already observed within each group. Under matched evaluator, comparison protocol, group size, prompt schedule, and query budget, it improves over Pairwise-GRPO across controlled comparisons, transfer settings, and blind human evaluations, suggesting that explicitly modeling preference cycles can lead to more stable and effective training for video generation models.