The widely used LIBERO benchmark may overstate how well robotic manipulation policies generalize. To address this, the authors introduce LIBERO-Pro2.0, a benchmark designed to systematically evaluate robustness under distribution shifts rather than just nominal task performance.

The benchmark spans 22 static perturbations and 20 runtime interventions across six dimensions: Condition, Environment, Observation, Execution, Robot State Configuration, and Language. This yields 1,648 validated shifted task instances. The authors also collected 39,390 training trajectories from model-generated and human demonstrations, and every shifted instance went through six rounds of iterative human review to confirm physical plausibility and task validity.

They evaluated 17 representative policies, including vision-language-action models, world-action models, and robustness-oriented policies. Under runtime distribution shifts, the models showed an average performance drop of 26.5 percentage points, with individual drops reaching 32.5 points. These results highlight a clear gap between nominal benchmark performance and robustness under real-world variations, underscoring the need for more systematic robustness evaluation in embodied AI.