RTL Repair Benchmarks: What Do They Actually Measure?

RTL repair benchmarks are meant to test whether a model can fix broken register-transfer-level code given a specification. The authors of this paper build a new infrastructure to check whether such benchmarks actually do that. They release RTLRepairDataset, with 5,784 (broken, fixed) training pairs, and RTLRepairBench, 459 held-out cases split into 374 syntactic and 85 behavioral mutations, plus a harness supporting both simulation and formal equivalence checking.

Using this infrastructure, they identify two measurement confounds. First, in one evaluated cohort, simulation and formal scoring give opposite signs for the apparent effect of providing a specification. Reset-aware rescoring attributes almost all of that disagreement to reset and first-cycle comparison semantics. Second, across six models and 274 mined failures, the difference between a specification-only baseline and a specification-conditioned recovery point estimate ranges from -8.5 to +18.3 percentage points, meaning a conventional repair score may not isolate whether the model actually used the faulty RTL.

The authors translate these findings into concrete evaluation controls: report initialization, reset, first-cycle, and unknown-value semantics; disclose oracle coverage and undecided cases; and include a specification-only baseline. The resources and audit are released to support behavioral repair benchmarking and diagnosis of what contributes to measured scores.