Grading law school exams is notoriously difficult because answers are open-text: case solutions, essays, and memos that apply law to facts. This makes assessment costly, and human graders frequently disagree with one another. A new paper on arXiv takes a systematization-of-knowledge approach to this problem, focusing on whether large language models can serve as reliable rubric judges.
The authors frame the task as one of trustworthiness: can LLMs grade consistently and fairly enough to replace or assist human graders? The abstract highlights the core issues—expense and grader disagreement—and points to LLMs as a possible solution, but the paper appears to survey existing research rather than present new experimental results.
Because the abstract is truncated, the full scope of the review is not clear from the source alone. Still, the significance is evident: if LLMs can grade legal answers with acceptable reliability, they could reduce costs and improve consistency in legal education assessment. The paper is a single source, so no comparison with other work is possible here.