A new arXiv preprint argues that current robot manipulation benchmarks cannot be trusted: leaderboards are either closed, simulated, or vulnerable to gaming. To address this, the authors propose HATCH, a robot escape room designed to make capability claims verifiable through physical, sensor-scored tasks.
HATCH is described as openly specified and physically replicable, meaning others could build the same setup. The key innovation is the use of hardware-attested task chains, where sensors confirm that a robot completes a sequence of manipulation steps in the real world. This is intended to prevent the shortcuts and cherry-picking that plague existing evaluations.
The paper is a single source, so there is no independent comparison to other methods here. Still, the proposal directly targets a known problem in robotics: the gap between reported performance and reproducible, trustworthy evaluation. Whether the escape room format will be adopted remains to be seen, but the emphasis on physical attestation is a concrete step toward harder, more meaningful benchmarks.