Reward learning from human preferences usually rests on a strong assumption: that the feedback a person provides captures everything relevant to the task. In practice, humans have limited attention and tend to teach by highlighting only certain features, leaving the rest of the reward specification implicit. That mismatch can produce reward models that are underspecified and fail to reflect the user's true intent.
The new paper, posted on arXiv, frames this as a problem of asking better questions. Instead of passively accepting whatever preferences a human volunteers, the authors propose a targeted active preference learning approach. The idea is to select queries that specifically probe the under-specified parts of the reward, forcing the human teacher to reveal aspects they might otherwise omit.
While the abstract does not detail the exact algorithm, the significance is clear: aligning AI systems with human values requires not just more data, but more strategic data collection. By acknowledging that human teaching is inherently partial, this work points toward reward learning methods that are more robust and better matched to how people actually communicate preferences.