Time series foundation models can take covariates as in-context inputs, which in principle lets them answer "what if" questions about future events. A new benchmark, WhatIf-Bench, asks how reliably they actually do this. The benchmark builds 37,665 tasks from mechanistic simulators of data centers, power grids, road traffic, and water networks, with controlled relationships between covariates and targets. Each task gives a history of events that intervene on covariates, then asks the model to forecast a future event that either replays seen values or sets them to unseen ones.

Across time series foundation models, frontier LLMs, and simple regression baselines, the authors find that LLMs achieve the highest skill, measured as improvement over a naive baseline. On "Replay" futures, all models do reasonably well, with skill scores between 48% and 85%. On "Novel" futures, where covariates take values not seen in history, scores fall to between -7% and 50%, and regression baselines drop the most. Performance on Novel futures also declines as events set more covariates jointly, while having more historical events helps all models—but large gaps remain.

The results suggest that interpolation to familiar scenarios is not the main difficulty; the hard part is generalizing to genuinely new combinations of covariate values. WhatIf-Bench is intended as a testbed toward arbitrary scenario analysis, and the authors make a leaderboard and code available.