Retrieval-augmented generation (RAG) systems are only reliable if they know when to say "I don't know." A new arXiv paper introduces DeclineCraft, a benchmark designed specifically to measure this ability. Instead of scoring only correct answers, it checks whether a model can recognize when neither retrieved documents nor its own parametric knowledge can support a response.
The benchmark is described as dynamic, which likely helps prevent models from memorizing test instances. It jointly evaluates retrieval and parametric knowledge, meaning a model must handle cases where the retrieved context is insufficient, its internal knowledge is insufficient, or both.
Because the source is a brief abstract, details on the benchmark's construction and evaluation results are not yet available. The work points to a growing focus on abstention as a core capability for trustworthy RAG systems.