ExeBench is one of the most widely used benchmarks for evaluating LLM-based decompilers, but a new arXiv paper argues that it has serious fidelity problems. The authors analyze ExeBench and identify nine issues that affect how faithfully a test measures a decompiler's actual ability.
The abstract reports that three of these issues are especially damaging, preventing LLM decompilers from being meaningfully assessed. The paper introduces ExeBench-FHD as a benchmark variant intended to address these problems and make decompiler evaluation more trustworthy.
Because the source is a single arXiv abstract, the full details of the nine issues are not available here. The key takeaway is that even popular benchmarks can quietly distort model evaluations, and the proposed fix aims to close that gap.