Retargeting-based approaches to learning manipulation from human video convert wrist and finger poses into robot commands using geometric mappings. These mappings need recalibration for each new robot and can pass hand-tracking errors into the control pipeline. EgoJEPA instead learns embodiment-agnostic representations directly from egocentric video.
EgoJEPA has three stages. First, it pre-trains on human actions with JEPA-style visual prediction conditioned on hand-motion differences, learning latent actions tied to effects. Second, a world model jointly predicts future visual dynamics and latent actions, separating human prior learning from robot-specific control. Finally, lightweight adapters tune the model for a particular robot.
Experiments covered five real-world manipulation tasks across three robot embodiments. Under out-of-domain evaluation, EgoJEPA reached a 74% average success rate, compared with 65% for a retargeting baseline. It also reduced cumulative cross-embodiment wall-clock cost by 47.2%, from 743.2 to 392.7 hours.