Text-to-video models can produce clips that look realistic but break the physical rules implied by the prompt. Existing benchmarks tend to assign holistic plausibility scores, which obscure exactly which behavior failed and why. PhyMetric, a new benchmark from researchers accepted at NeurIPS 2026, recasts the problem as verifiable scene-level question answering.
The benchmark contains 1,115 scenes spanning 50 physical categories across six domains, paired with 8,920 binary questions over 12 evaluation dimensions. Each scene comes with eight targeted yes/no questions and a structured physical prior that specifies the governing law, expected temporal sequence, observable signatures, and common failure modes. At evaluation time, a vision-language model judge answers each question from question-aware sampled frames conditioned on that prior, and correctness is aggregated hierarchically into scene, dimension, category, and model scores.
Experiments with four open-source text-to-video models and three VLM judges show that the targeted-question format is essential: when each targeted question is replaced by a single generic plausibility query, evaluator accuracy falls to chance. The structured priors provide a complementary signal that improves alignment with human judgments, although the size of that improvement depends on the capacity of the VLM judge. The authors position PhyMetric as a way to turn opaque holistic scoring into interpretable, scene-level inspection of physical plausibility.