Interpreting video-understanding models requires knowing which moments in a clip actually matter for a prediction. A new arXiv paper argues that existing attribution methods are poorly suited to this task: they tend to spread evidence diffusely across many near-duplicate frames, obscuring the decisive moments.
The paper's title points to a proposed remedy: sufficient frame subsets. Rather than attributing a prediction to a broad swath of frames, the idea is to identify a minimal subset of frames that is sufficient for the model's decision. This would shift the focus from diffuse evidence to a compact, decision-relevant set.
If successful, such an approach could make video models easier to interpret and debug. By clarifying when a model makes its critical observations, sufficient frame subsets could give researchers a sharper tool for understanding what these systems are actually looking at.