Human video demonstrations are a rich source of manipulation strategies, but videos generated from simulated scenes often contain geometric distortions, inaccurate contacts, and other physical inconsistencies that make the recovered motion difficult to execute on a robot. The Gen2Dex framework is designed to convert such simulation-grounded videos into physically executable dexterous robot demonstrations.
The perception pipeline exploits the known scene through model-based object tracking with dense foundation-model features and depth-based hand-object alignment. The recovered human motion is then retargeted and physically grounded using residual reinforcement learning. A backward curriculum and a cache of physically valid reset states allow the policy to correct imperfect video-derived references while preserving the demonstrated manipulation strategy.
On clean ARCTIC demonstrations, the method is competitive with prior dexterous retargeting approaches and shows greater robustness when references are extracted from generated videos. The authors demonstrate diverse rigid and articulated manipulation tasks in simulation and report successful open-loop transfer of the optimized trajectories to a real dexterous robot.