Sentiment analysis benchmarks often rely on a single annotator per sentence, meaning the test labels can carry that person's idiosyncratic judgments. The authors of a new arXiv paper argue this is a common but underappreciated weakness in evaluation data.

To address it, they introduce NEPSEE, a Nepali sentiment benchmark built from two annotators who labelled the same 5,614 sentences independently and without seeing each other's work. This design is intended to reduce the influence of any one individual's conventions on the final labels.

The abstract does not report inter-annotator agreement scores or describe the text domain, so the practical effect of the double-annotation approach is not yet quantified. Still, the resource offers a concrete step toward more reliable sentiment evaluation for Nepali, a language that remains underrepresented in NLP benchmarks.