A new preprint on arXiv introduces LintEval, a benchmark for evaluating large language models on linter-guided file refactoring. Linters are tools that flag code for performance, style, error-prone, and other issues, and they are widely used in code maintenance and real-time IDE editing. The authors note that existing code benchmarks do not explicitly cover this kind of task.
The benchmark is designed around the idea that refactoring often involves responding to lint alerts, not just generating code from scratch. By focusing on file-level changes, LintEval aims to capture a realistic maintenance scenario. The abstract does not yet describe the benchmark's scale, the models tested, or the results.
Because this is a single preprint, there are no other sources to compare against. The contribution is a proposed evaluation method; whether it becomes a standard reference will depend on the full paper and community adoption.