A new arXiv paper introduces CETBench, a benchmark designed to test whether large language models can determine if two programs are functionally equivalent. The authors position this as a core problem for tasks like code rewriting, where a model must preserve behaviour while changing the implementation.
The abstract states that the team studies how well LLMs can judge equivalence between programs. The paper's title goes further, characterising LLMs as "brittle" to simple code transformations, suggesting that even minor changes can trip up current models.
Because the source is only the abstract, specific results and experimental details are not yet available. The significance lies in the benchmark itself: it offers a way to measure a capability that is often assumed but rarely tested directly. The authors note that benchmarking code equivalence helps assess LLM performance in code-related tasks, though the full scope of that claim remains to be seen in the paper's body.