Large language models are often proposed as a way to automate the labour-intensive marking of student writing. But according to a new arXiv preprint, most existing evaluations of this idea are narrow: they typically test a single model, a single prompt, and a single institutional context. The authors argue that this limits what we can conclude about how well LLMs actually match human markers.

The study, described in the abstract, examines how machine–human agreement in marking undergraduate assignments changes when the model, prompt, or institution is varied. The finding is that agreement is specific to these factors rather than a stable property of the LLM itself. In other words, a system that agrees closely with human markers in one setting may not do so in another.

The abstract does not provide full details of the experiments or numerical results, so the precise size of these effects is not yet available. But the implication is clear: claims about LLM grading accuracy should be treated as context-dependent unless tested across a wider range of models, prompts, and institutions. The authors appear to be cautioning against overgeneralising from single-setting evaluations.