Automated essay scoring (AES) with large language models has been proposed as a practical tool for language assessment, but high-stakes testing demands more than just plausible scores. A new paper on arXiv (2610.25724) takes a mechanistic look at whether LLM-based scorers actually follow rubrics, studying 25 base models to find evidence of rubric control.
The paper's abstract frames the core tension: while LLM-based AES offers promising use cases, high-stakes testing requires scores that are both controllable and verifiable. The authors report mechanistic evidence on rubric control, suggesting they examined internal model behavior rather than only final outputs. However, the abstract is cut off, so the full findings and methodology are not available from the source alone.
What is clear is that the paper positions current LLM-based scorers as potentially falling short of the controllability and verifiability needed for consequential decisions. This adds to a growing conversation about the reliability of LLMs in assessment contexts, though the specific limitations and proposed remedies would require reading the full paper.