Long-horizon terminal benchmarks are designed to test whether an AI agent can keep making meaningful progress through a long, multi-step workflow. The newly announced Long-Horizon-Terminal-Bench 2.0 shifts the focus from a simple pass/fail on the final output to a more granular evaluation of how the agent behaves along the way.
According to the paper's abstract, the benchmark uses process rewards to measure sustained progress, rather than only checking whether the final state matches an expected result. This approach is intended to capture whether an agent can maintain coherent, incremental advancement over an extended task, which is a more realistic and demanding evaluation than a single binary outcome.
The new version also emphasizes automating the construction of diverse tasks, which should make it easier to generate a wide range of long-horizon challenges without manual effort. The preprint is available on arXiv under identifier 2610.14146.