Machine unlearning for multimodal large language models (MLLMs) is advancing rapidly, and with it the need for rigorous evaluation. A new preprint on arXiv asks whether current benchmarks actually measure what they claim. The title poses the question directly: "Are We Really Measuring Forgetting?"

The authors argue that existing benchmarks largely inherit evaluation protocols from earlier settings rather than being designed for the multimodal case. This raises the risk of pitfalls that make reported unlearning results look better — or worse — than they are. The abstract, which is truncated, does not spell out the specific pitfalls, but the central warning is clear: evaluation methodology has not caught up with the models it is meant to assess.

If unlearning benchmarks do not genuinely measure forgetting, then claims about removing sensitive or copyrighted content from MLLMs rest on shaky ground. The paper is a reminder that progress in unlearning is only as credible as the metrics used to verify it.