Large language models are increasingly used for discrete operational decisions—whether to flag a phishing email, escalate a ticket, or approve a transaction. These choices typically require balancing speed against accuracy. A new preprint from arXiv introduces System1-Arena, a framework designed to systematically test which of these decisions can be handled by 'System 1' models—those that return probabilities over options in a single forward pass—rather than slower, more deliberative 'System 2' reasoning.

According to the abstract, the arena evaluates models such as Jev and the OpenAI Decisions API, which are built for fast, probabilistic classification. The key question is not whether these models can make decisions at all, but where their reliability boundaries lie. By pitting them against more compute-intensive approaches, the benchmark aims to map out a cost-performance frontier for decision-making.

The significance is practical: if a fast model can match a slow one on a given task, organizations can save latency and compute. If not, the decision should be escalated. The paper is a single source, so there are no conflicting findings to compare, but the framing suggests a growing effort to formalize when 'good enough' fast inference is truly good enough.