A common way to evaluate LLM-driven evolutionary search is to count how many candidate programs are proposed before finding a solution. A new preprint on arXiv argues that this metric is misleading, because it ignores the varying number of tokens each proposal costs to generate. The authors demonstrate that proposal counts hide the true expense of search and redo the comparison using token counts.
By replaying 600 logged runs, the paper shows that token-based accounting reveals costs that proposal-based metrics miss. The authors do not simply tweak the numbers; they re-run the comparison from the logs, allowing a direct look at how the two metrics diverge. The implication is that conclusions about which search strategy is most efficient may depend heavily on whether the yardstick is proposals or tokens.
The study is a single-source contribution, so there are no conflicting findings to weigh. Its main recommendation is clear: future benchmarks for LLM evolutionary search should report token consumption alongside proposal counts, or risk comparing strategies on a metric that does not reflect the real computational burden.