Contact-rich manipulation is hard to model because robots need data showing how contact forms, slides, and releases under different forces. Collecting that data by teleoperation is slow and often lacks force feedback. The authors of ACTS ask whether such dynamics can be learned from human physical interaction instead, then transferred to a robot.
ACTS is a visuo-tactile world model trained only on human interaction data. Operators moved handheld GelSight sensors while the system recorded third-person and wrist video, tactile images, 6-DoF motion, and normal force. This produced React, a 4.9-hour bimanual dataset covering three objects. Actions are represented as sensor motion plus force change, giving a shared action space for both human collection and robot execution.
On held-out human data, ACTS improved scene-view prediction by 0.7–2.0 dB PSNR over the stronger of two baselines trained from scratch on React. In robot teleoperation rollouts without fine-tuning, predicted tactile imprints appeared and disappeared with recorded contact. Visual transfer was less robust: camera predictions beat frame copying on PushT, were mixed on Rope, and fell below it on Motherboard, while wrist views sometimes blurred or hallucinated structure. The authors conclude that contact dynamics transfer more reliably than visual appearance from human to robot.