Object counting in videos remains a poorly characterized capability of multimodal large language models. Although counting questions are common in existing VideoQA benchmarks, the ground-truth count distributions in those datasets raise concerns for evaluating this skill. A new paper introduces CountVideoBench, a benchmark built to evaluate prediction biases in object counting.
By focusing specifically on counting, CountVideoBench aims to provide a more reliable measure of how video-language models behave when asked to enumerate objects. The authors argue that existing benchmarks are insufficient for this purpose, and they offer CountVideoBench as a targeted tool for exposing and quantifying counting biases.