Video world models are often evaluated with a single memory score that blends several distinct abilities. According to a new arXiv paper, this makes it hard to tell whether a model failed to remember the initial scene, failed to execute the requested change, or failed to render the result in the final frame.

The authors argue that a proper memory test should answer three separate questions: whether the scene was set correctly, whether the asked change was actually carried out, and whether the last frame reflects that change. By folding these into one number, current tests leave little room for diagnosis when a model scores poorly.

The paper's proposed decomposition is a step toward more interpretable evaluation. Instead of asking "does the model remember?", it asks where memory breaks down. This could help researchers target specific weaknesses in video world models rather than treating all failures as a single problem.