Chemistry evaluation has moved beyond multiple-choice probes toward open-ended expert problems, but the benchmarks that define the current frontier still score answer endpoints alone, according to a new arXiv paper. The authors argue that such scoring conflates memorized facts with genuine reasoning, since a model could produce a correct final answer without demonstrating the step-by-step logic expected in a laboratory setting.

To address this, the paper introduces ChemArgus, a frontier benchmark designed for lab-grade multimodal chemistry reasoning. It proposes automated scientific diagnostics that assess how models arrive at their answers, rather than only checking whether the endpoint is correct.

The abstract is brief, so details of the benchmark's construction and evaluation methodology are not yet available. But the motivation is clear: scoring the reasoning process itself, rather than only outcomes, could offer a more meaningful measure of progress in AI-driven chemistry.