Word error rate (WER) treats every substitution, deletion, and insertion as equally costly, even though some errors barely matter and others completely change the intended message. Whole-utterance semantic scores, meanwhile, can say something is wrong without pinpointing which part of the transcript caused the problem. A new paper from researchers at Shanghai Jiao Tong University and colleagues introduces SMART, a framework that tries to make ASR evaluation both semantic and traceable.
SMART first groups the differences between a reference transcript and a hypothesis into coherent segments, then applies a structured contextual rubric to assign explicit consequence labels. These labels feed a new metric, Semantic Impact Error Rate (SIER), which reports consequential errors at the utterance, chunk, and lexical levels. In tests against majority-vote human labels, SMART reached 0.843 F1, compared with 0.745 for direct LLM judgment, and the authors report that results stayed relatively stable across different LLMs used for evaluation.
The paper also describes a target-reference-free rubric for ASR correction, which uses the target hypothesis and surrounding ASR hypotheses to guide fixes. This rubric outperformed a matched no-rubric baseline on both datasets, yielding absolute reductions of 0.85–0.98 percentage points in MER and 1.18–1.65 percentage points in SIER_lex. The authors argue that MER and SIER capture different aspects of ASR quality, suggesting that semantic impact should be measured alongside traditional error rates.