A new arXiv paper takes aim at the way time series foundation models (TSFMs) are evaluated. It argues that static public benchmarks, which are widely used to rank models, suffer from a fundamental flaw: their test distributions are fixed and narrow. Because the same benchmarks are reused over and over, researchers can tune models to perform well on those specific tests rather than to generalize broadly.
The paper suggests that reported performance on these leaderboards often reflects adaptation to the benchmark itself, not true model capability. To address this, the authors propose a shift toward capability-focused evaluation that incorporates structural diversity. This would mean testing models across a wider range of data structures and conditions, rather than relying on a single static set of tasks.
While the abstract does not provide implementation details, the core message is clear: the field needs evaluation methods that measure what models can actually do, not just how well they have memorized a particular benchmark. The proposal aligns with growing concerns about benchmark overfitting across machine learning, though it applies these concerns specifically to the time series domain. No other sources were provided, so this summary reflects only the paper's own claims.