Code produced with LLM assistance may contain subtle defects, including API hallucinations—calls to nonexistent or incorrect interfaces. Conventional rule-based metrics only score code superficially and may miss such issues, motivating the need for better evaluation methods before deployment.
The arXiv paper "Benchmarking Code LLM-as-a-Judge via Constrained Perturbation and Calibration" proposes a benchmarking approach that applies constrained perturbations to code samples, creating controlled test cases to assess how well LLM judges detect introduced flaws. The method also incorporates calibration to align judge scores with human expectations, aiming to make LLM-based evaluation more trustworthy.
While the abstract does not report specific results, the contribution lies in the benchmarking methodology itself. By systematically perturbing code and calibrating judge outputs, the work offers a way to measure and improve the reliability of LLM-as-a-judge systems for code quality assessment.