Latent world models learn compact, action-conditioned state transitions, but planning with them usually requires expensive test-time optimization over action sequences. Researchers propose replacing that online search with a goal-conditioned inverse model that directly predicts actions from current and goal observations. The challenge is that simple action prediction does not guarantee that goal representations encode action-relevant information or that predicted actions align with the learned latent dynamics.

The new framework, Hindsight Inverse JEPA Planning (HIJP), addresses both issues during training. It relabels future observations along offline trajectories as goals, providing action supervision across multiple horizons. By retaining gradients through the goal representations, this supervision directly shapes the visual representation. HIJP also feeds predicted actions into the latent predictor and aligns the resulting next-state representations with encoded successor observations, enforcing action-transition consistency.

At inference, the inverse dynamics model and latent predictor are applied alternately to generate multi-step action sequences without any online search. Across four visual control tasks, HIJP achieves 86.8% overall success and a 1.9–16.5x planning speedup over search-based planning, suggesting that search-free latent planning can be both faster and more accurate in practice.