Video world models are good at producing realistic frames, but they struggle to stay consistent over long sequences because they only see partial views and must remember how past actions changed the world. The authors of StateCraft argue that remembering raw observations is not enough: after a series of actions, an old observation may no longer reflect the true state of the environment.
To fix this, StateCraft introduces a persistent abstract state that tracks entities and their changes as the world evolves. This abstract state is updated by a language model, while a video diffusion model uses it to condition the visual output. Because the abstract state is explicit, it can be edited directly, and the model can follow edits to states it has never seen in training.
The authors built a data collection pipeline that generates densely annotated trajectories without human or agent supervision, and they evaluate on generative interfaces—a challenging setting where a single action can alter parts of the state that are not currently visible. StateCraft outperforms state-of-the-art baselines on their benchmark, maintaining high self-consistency and strong generation quality over long trajectories. A proof-of-concept 3D experiment, where actions change the illumination of an unobserved room, suggests the paradigm also extends to spatial environments.