Vision-language-action policies are usually judged by a single number: how often the whole task succeeds. According to the new arXiv preprint RoboCasa-Shift, that aggregate score hides important detail. It does not show how individual executions change when the environment shifts, nor where in a long-horizon task the failures actually occur.
The paper introduces RoboCasa-Shift as a stage-aware robustness evaluation for long-horizon household manipulation. Instead of treating a task as one monolithic outcome, the proposed evaluation tracks performance through stages, so that robustness can be assessed at the point where failures emerge.
Because the abstract is brief, the exact benchmark design and results are not available from this source. The central claim is clear, though: aggregate task success is not enough to understand VLA robustness in household settings.