Robots operating in the real world face a problem that standard world models ignore: many factors that determine how the world responds to an action are invisible. Friction, mass, and even human-defined rules are not directly observable, yet they shape every prediction. The T2-Z method, described in a recent arXiv preprint, addresses this by giving a world model a compact, learnable "environment latent" that conditions its predictions.
During training, the approach alternates between optimizing the world model with latents fixed and optimizing the latents with the model frozen. This lets latent structure emerge purely from what is needed to predict outcomes. At test time, the model stays frozen and a newly initialized latent is optimized from one or more observed interactions, allowing the robot to adapt to previously unseen environmental configurations.
The authors report results across seven simulated settings and four real-world robotic tasks. Object prediction error fell from 15.20 to 7.00, and action success rose from 33.73% to 72.96%. Ablations show that even a single test-time interaction trajectory can be enough to infer the latent efficiently.