Open-vocabulary human-object interaction (HOI) detectors output free-text descriptions of interactions, which makes evaluation tricky. According to a new arXiv paper, the standard approaches have complementary flaws.
Category scoring, which matches predictions against a fixed set of labels, rejects valid paraphrases that convey the same interaction. Semantic similarity scoring, on the other hand, can reward predictions that are not actually correct, because it relies on loose textual overlap.
The paper argues for auditing both semantic and relational reliability in HOI evaluation, suggesting that current metrics may not faithfully reflect detector performance. The abstract does not provide specific solutions, but it underscores the need for more robust evaluation methods.