Existing benchmarks for visual world knowledge in multimodal large language models (MLLMs) are often narrow in category coverage and allow models to answer correctly using coarse-grained cues or elimination strategies. To address this, the authors introduce VisualWorldBench, a taxonomy-grounded multi-task benchmark designed to force precise discrimination among fine-grained visual entities.
The benchmark is built on a hierarchical taxonomy of 21,291 leaf-node classes across 8 visual domains, with 32,541 evaluation instances whose ground-truth labels span 15,479 categories. It includes four complementary tasks—closed-set recognition, contrastive selection, precise localization, and open-world naming—which vary in answer openness and knowledge retrieval direction. Most questions are constructed from taxonomically proximate categories, making it difficult for models to rely on vague associations.
Experiments show that the strongest closed-source model achieves only 80.4% average accuracy, exposing capability gaps hidden by conventional benchmarks. The authors also construct VisualWorldBench-50K, a training resource covering over 50,000 categories. Fine-tuning on this dataset yields more than 21 percentage points improvement on VisualWorldBench, with consistent transfer to public fine-grained recognition and visual knowledge benchmarks.