HexaShell, described in a recent arXiv preprint, is a deliberately thin harness for testing how far vision-language-model (VLM) agents can go when they control a robot rather than a computer. The agent operates a two-armed robot through a command line, seeing three calibrated RGB cameras. After a failed attempt it reads the log, diagnoses the problem, and writes or fixes small programs—tools—that it keeps as code, analogous to a coding agent fixing a failing test.

The tools rely only on what the robot can observe. To locate objects, the agent uses triangulation: if two cameras see the same point, the point is where the rays meet; if a point lies on a known plane, such as the table or a block top, it is where the ray meets that plane. When the head camera is insufficient, the robot moves a wrist camera to get a second view. The authors build tools on ten layouts of one seed per task, freeze them, and test on ten new layouts of the same seed, using Codex with GPT-6-Astra at medium reasoning effort.

Across all 42 RoboDojo tasks, HexaShell reaches a 39.7% success rate and a progress score of 48.0, averaged as on the leaderboard—a new state of the art. The breakdown is telling: pushing reaches 92.5% and memory tasks 65.0%, but tight insertion only 8.3%. The agent usually sees the failure but cannot make the motion work, suggesting the limit is contact rather than reasoning. The authors open-source the tools evolved so far.