Automated relevance judgment is a bottleneck in information retrieval evaluation, particularly for languages with fewer resources. A new arXiv preprint proposes using Jev, a System One model designed for fast, structured decision-making, as an automated judge for Brazilian Portuguese collections. The authors use Jev-as-a-judge to generate relevance judgments across four IR collections, covering both relevance assessment and reranking.

The abstract describes the setup but does not report results, so it is unclear how well Jev's judgments align with human annotations or with judgments from larger models. The full paper presumably includes those comparisons. If Jev proves reliable, it could offer a faster, lower-cost route to evaluating Portuguese-language IR systems; if not, the study still documents a systematic attempt to apply a System One model in this role.