Vision-language reward models are typically trained on annotated preference pairs, which are expensive to collect and can disagree. A new arXiv preprint describes a reward model built without any human preference labels. The authors write that it uses a linear head on a representation, though the abstract cuts off before specifying the exact backbone.
The motivation is to sidestep the cost and inconsistency of human annotations. The abstract also notes that existing reward models are ranked by benchmark accuracy, a separate evaluation criterion. By removing the need for human labels, the method directly addresses these issues.
Because this is based on a single abstract, details about training data, evaluation, and results are not yet available. The key claim is that a reward model can be built without human preference labels, but the full methodology remains to be seen.