Evaluating LLM Code: Translation Benchmarks and Metric Validity

Three recent arXiv papers converge on a common concern: evaluation in LLM-based software engineering is only as trustworthy as the tools used to measure it. Each takes a different slice of the problem—translation, generation, and quality metrics—but together they argue that current benchmarks and metrics need closer scrutiny.

CodeTransBenchmark (arXiv:2609.20257) targets LLM performance in translating and repairing code across languages, suggesting that existing tests may not capture real-world failures. CodeAssay (arXiv:2608.03535) goes further, proposing a multi-metric benchmark with audited ground truth, because test-based correctness alone can miss important aspects of code quality. The third paper (arXiv:2609.20411) steps back from LLMs entirely and examines code quality metrics themselves, finding that their construct validity—whether they actually measure what they claim—is rarely evaluated.

The three studies differ in scope: one focuses on cross-language translation, another on generation benchmarks, and the third on metric design. Yet they share a core message: without reliable references, tests, and metrics, claims about LLM coding ability remain fragile. The papers also differ in remedy—one builds a new benchmark, one adds audited ground truth and multiple metrics, and one calls for validating the metrics themselves.