Text-to-video retrieval models are usually judged on whether they find the right clip, not on whether they understand what changed inside it. A new benchmark, GenState-AI, is designed to expose that gap by testing whether models capture state changes and final outcomes rather than matching static appearance. The dataset contains 956 AI-generated videos and 195 human-verified state-change queries across four domains: Toy, Cartoon, Home, and Manipulation.

The benchmark was built using prompt-controlled variants that either create conflicting temporal or state outcomes or swap in semantic substitutions. After human verification, all valid videos contribute to subset-level retrieval pools, while complete query-specific triplets support diagnostic hard-negative ordering. This design lets researchers separate global retrieval quality from fine-grained temporal and semantic grounding.

When the authors evaluated VCF-Lik, Qwen3-VL, and LanguageBind, they found that Qwen3-VL (8B) had the best macro-average Recall@1 and triplet consistency, while LanguageBind achieved the highest semantic discrimination accuracy. However, stronger retrieval did not consistently lead to correct hard-negative ordering. Notably, Qwen3-VL and LanguageBind were weak on Manipulation triplets, where VCF-Lik performed substantially better, suggesting that different models excel at different aspects of video understanding.