The paper argues that existing evaluations of LLM-based code optimization rely too heavily on end-to-end speedup, which can obscure whether a model actually performs the intended optimization. A model may appear to improve performance even when it has not genuinely applied the optimization logic it was asked to execute.

This issue is particularly severe for compiled languages, because the compiler itself can introduce transformations that yield speedups unrelated to the model's semantic changes. To address this, the paper proposes a pattern-driven evaluation framework designed to isolate what it calls "compiler-transcendent semantic reasoning"—that is, whether the model truly understands and applies the optimization rather than benefiting from compiler behavior.

By focusing on identifiable patterns rather than aggregate speedup, the approach aims to give a more direct test of an LLM's reasoning ability. The paper's contribution is primarily methodological, offering a way to evaluate code optimization that is less confounded by the surrounding compilation pipeline.