Evaluating and refining robot policies through physical interaction is slow and expensive, which has motivated action-conditioned world models that act as interactive simulators. A key failure mode, however, is that predictions can look visually plausible while misrepresenting contact dynamics, giving unreliable feedback. PhysCanvas, introduced in a new arXiv preprint, is designed to close that gap by unifying vision and touch in a single state-consistent model.
PhysCanvas represents multi-view RGB, tactile images, marker displacement, task state, and actions in a unified vision space, sharing a video VAE and a pretrained generative backbone. Motion images derived from forward kinematics and camera projection provide spatially aligned action conditioning, while an initial RGB anchor and updated reference observations guide sequential generation. During post-training, three frozen reward models supervise scene preservation, visual-tactile compatibility, and agreement with real task outcomes.
Across four real-world manipulation tasks, the authors report that State Reward Model supervision increases the arithmetic mean of task-wise Pearson correlations between world-model and real-world success rates from 0.602 to 0.885. Policy post-training in the reward-guided world model also improves π0.5's mean real-world success rate from a 59.1% supervised fine-tuning baseline to 70.8%.
The results suggest that unified multimodal modeling and consistency-aware post-training can make world models reliable enough for closed-loop policy rollouts and reward-guided learning, reducing the need for costly physical interaction in robot manipulation.