Vision-language-action models (VLAs) have shown high success rates on manipulation tasks they were trained on, but a new paper argues that these numbers may not tell the whole story. The authors introduce VLA-GAUGE, a framework designed to systematically evaluate whether such models genuinely generalize to unseen situations and remain robust to perturbations.

The motivation is that existing benchmarks may overstate real-world capability. By focusing on zero-shot generalization and perturbation robustness, VLA-GAUGE aims to provide a more rigorous test of what VLAs actually learn, rather than how well they memorize familiar scenarios.

The abstract notes that current success rates leave open the question of whether they reflect genuine generalization. VLA-GAUGE appears to be a response to that gap, though the full methodology and results are not detailed in the provided text.