A new arXiv paper, "Decoupling Knowledge Forgetting and Refusal Learning in MLLM Unlearning," argues that standard evaluations of multimodal unlearning are too shallow. The authors note that current tests typically only check whether a model no longer outputs the target information, which they call a surface-level criterion.
This criterion, they say, masks two critical failure modes. One is that knowledge suppression techniques may appear successful while the underlying knowledge remains intact, and the other is that the model may simply learn to refuse to answer rather than actually forget the information.
The paper's title suggests a solution: separating knowledge forgetting from refusal learning in both evaluation and training. This distinction, the authors imply, is necessary to build unlearning methods that genuinely remove information rather than just hide it.