Passive human videos of everyday activities contain a vast amount of experience that robots could exploit, but this resource is unstructured and difficult to use directly. Traditionally, robots require demonstrations for each specific task, which is costly and limits scalability. MoCaP (Mobile Code-as-Policies) addresses this by recovering actionable knowledge from such unstructured videos, enabling robots to act without per-task demonstrations.
The approach builds on the code-as-policies paradigm, where robot policies are expressed as computer code generated from language and perception. By applying this to passive human videos, MoCaP aims to achieve zero-shot mobile manipulation—meaning the robot can handle new tasks it has never seen in training. This could significantly reduce the need for task-specific data and allow robots to learn from the abundance of human activity footage already available.
The abstract notes that the key challenge lies in recovering knowledge from unstructured sources, which MoCaP tackles directly. While the abstract does not provide experimental details, the proposed direction suggests a promising path toward more general and data-efficient robot learning.