LLM Evaluation Expands: New Benchmarks and Peer-Review Evidence

The latest wave of arXiv preprints suggests that evaluating large language models is no longer just about leaderboard rankings. The six papers here take different routes: some introduce new benchmarks, others probe existing practices. But they share a common thread: deciding what to measure—and how—has become as important as the models themselves.

Three of the papers offer new evaluation tools. PetriBench targets reasoning over dynamic state spaces, designed to be compact and self-contained, unlike benchmarks that isolate skills or depend on external knowledge. Another study asks whether classical models still deserve training on tabular data when LLMs can label rows from plain-English descriptions—a capability already appearing in spreadsheet tools. BuildBench, meanwhile, tests LLM agents on compiling real-world open-source software, a task previously handled by manually curated rules.

The other papers look at evaluation from different angles. One randomized experiment at ICML 2026 examines how reviewers actually use LLMs and whether different policies affect review outcomes. Another paper highlights 'reasoning collapse' in LLM-based embedding learning, arguing that how reasoning quality is handled matters for producing useful embeddings.

Taken together, the sources agree that evaluation is a moving target, but they differ in what they emphasize: task coverage, deployment practicality, or human workflows. No single approach dominates, and the divergence itself is informative.