Building interactive benchmarks for AI agents is expensive, especially when the goal is to ground answers in verifiable sources. A new arXiv paper, BTS-AgentBench, proposes a pipeline that starts from the extensive local telemetry logs that operational facilities already accumulate, turning them into agent-ready evaluation tasks.

The approach is described as deterministic and replayable, meaning the same logs should yield the same benchmark tasks each time the pipeline runs. By relying on read-only telemetry, the pipeline avoids interfering with live systems while still producing evaluations whose answers can be traced back to the original data.

The abstract notes that site-specific measurement histories also play a role, though the full details are not available in the excerpt. As a single-source summary, this is a preliminary look at the method; the paper itself would be needed to assess how the pipeline handles the inherent messiness of real-world telemetry.