ZeroDex, described in a preprint by Jisoo Kim and colleagues, aims to give robots the open-world flexibility of vision-language models while supplying the metric 3D information manipulation actually requires. Instead of training a policy, the system uses a VLM to decompose a language instruction into atomic tasks and lifts 2D predictions from multiple calibrated RGB views into 3D contact points, placement targets, and tool-motion anchors.
Those 3D groundings drive a small library of reusable primitives covering pick-and-place and tool use. For tool use, the system retrieves an object-centric atomic action and aligns its stored 6D tool trajectory to the grounded keypoints; for dexterous grasping, it expands a lifted grasp keypoint into an affordance region and generates feasible grasp-motion pairs with an arm-hand motion generator.
In real-world experiments, the authors report improved 3D grounding accuracy and higher task success than zero-shot visual-prompting and constraint-based baselines that ground from a single RGB-D view. The demonstration-free system also outperformed VLA policies fine-tuned on 100 task demonstrations. The paper further demonstrates long-horizon manipulation, where execution verification and replanning recover from intermediate failures, and shows generalization to unseen objects and tool-use tasks in novel scenes.