Language models are increasingly used for forecasting, which raises the question of how well they actually model the world. A new arXiv preprint introduces CivWorldBench, a benchmark designed to evaluate world-model capabilities in structured multi-agent sandbox worlds.
The authors argue that evaluation must go beyond predictive accuracy. CivWorldBench therefore also tests hidden-state inference — whether a model can infer unobserved states — and intervention prediction, assessing how models anticipate the effects of actions.
The abstract is brief and the full methodology is not yet detailed in the available text. As a single preprint, the claims have not been peer-reviewed, and no comparison with other benchmarks is provided in the source.