Closed-loop evaluation is often considered a more faithful way to assess end-to-end (E2E) driving models than open-loop evaluation, because it tests the model's ability to react to the environment in real time. According to a new arXiv paper, Bench2Drive—built on the CARLA simulator—has become the dominant benchmark for this kind of evaluation.
The paper, titled "Comfortably Stuck: Evaluation Pitfalls in Bench2Drive's Closed-Loop Protocol," sets out to show that there are pitfalls in this protocol. The abstract's text cuts off after "We show…," so the specific findings are not available in the provided source. However, the title suggests that models may become "comfortably stuck"—perhaps meaning they can achieve good scores while failing to make progress, or that the benchmark rewards staying put rather than driving effectively.
Without the full abstract, readers should treat the exact nature of the pitfalls as unspecified. The paper's main contribution, as far as the source reveals, is to call into question the reliability of a widely used benchmark, even as closed-loop evaluation remains the preferred method for many researchers.