Google Research's latest algorithms post tackles a practical question in AI evaluation: how many human raters are needed to build a trustworthy benchmark. Since many benchmarks rely on human judgments to label or score model outputs, the size of the rating panel can influence whether a benchmark is stable and reproducible.
The post examines the trade-off between adding more raters and getting more reliable benchmark scores. It suggests that there is a threshold beyond which extra ratings add little value, making the goal not maximum raters but enough raters for consistent, defensible results.
This is a single source, so there are no conflicting findings to reconcile. The piece contributes to a broader theme in AI research: careful benchmark design matters as much as model capability.