Large language models sometimes need to selectively remove harmful or sensitive knowledge, a process known as unlearning. But according to a new arXiv preprint, current benchmarks measure this capability incompletely.
The abstract explains that existing benchmarks depend on disjoint forget and retain sets—meaning they test whether a model forgets one set of information while keeping another. This setup misses cases where the same knowledge should be forgotten in one context but retained in another.
To address this, the paper presents ConceptGuard, a benchmark specifically designed for context-sensitive unlearning. While the full methodology is not detailed in the abstract, the title and framing suggest a shift toward evaluating unlearning in a more nuanced, context-dependent way.
The work highlights a growing recognition that unlearning is not simply about erasing facts, but about managing when and where those facts should apply.