A new arXiv paper proposes a task for evaluating semantic relation knowledge in large language models. The task asks an LLM to predict the relation between two glosses, using a bidirectional relation prediction setup. The authors describe the task as challenging.
Semantic relations organise word senses and are fundamental to lexical semantics, the paper notes. By focusing on gloss pairs, the evaluation targets whether models can recover the connections that structure dictionary-style definitions.
This summary draws on a single source: the paper’s abstract. No experimental results or comparisons with other benchmarks are reported in the available text.