Robot world models often rely on robot actions as conditioning signals, but those actions are tied to a particular robot's mechanics and do not map cleanly onto visual changes. A new arXiv paper, TrAct, argues that this weak alignment limits how well actions can guide visual prediction.
The paper proposes using visual tracks—point trajectories in image space—as an alternative conditioning signal. Because visual tracks describe what actually moves in the scene, they may offer a more natural bridge between control and prediction, independent of the robot's specific embodiment.
The available abstract outlines the motivation and the proposed direction, but does not include experimental results. As such, the paper's claims remain to be validated, though the conceptual shift from actions to visual tracks is a notable idea for robot learning research.