Reinforcement learning agents that optimize purely for task rewards can settle on behaviors that are efficient but clash with human expectations. Preference-based RL tries to fix this by having humans compare trajectory pairs, but the resulting binary labels do not explain why one behavior is preferred. The authors of this preprint propose adding reason tags to each preference judgment, giving the reward model extra supervision about the rationale behind the choice.

The framework trains a human reward model on both the preference labels and the reason tags. During policy optimization, this learned human reward is combined with the environment reward, so the agent can keep improving task performance while also respecting human-aligned preferences. The authors argue that this reason-guided formulation improves the interpretability of the resulting policy compared with preference-only learning.

This is a single arXiv preprint, and no independent sources were available for comparison, so the claims should be read as preliminary rather than established results.