A new arXiv paper reports that recent multimodal large language models perform strongly on visual reasoning tasks, approaching human-level results on existing STEM benchmarks. The authors caution, however, that these capabilities do not hold up when models are faced with complex real-world scenes.

To address this gap, the paper presents Genesis, a framework built around what it calls visual augmentation primitives for multimodal chain-of-thought reasoning. The available abstract introduces the approach but cuts off before providing details on how these primitives work or how they were evaluated.

The work highlights a disconnect between benchmark success and real-world robustness. It also suggests that current STEM-focused visual reasoning tests may not fully capture the difficulties models encounter in less controlled settings.