Evaluating abstractive summaries is harder than it looks. Traditional overlap-based metrics like ROUGE measure surface similarity but often miss factual errors or semantic quality. A recent article by Eugene Yan breaks down the landscape of evaluation methods, grouping them into reference-based, context-based, and preference-based approaches, and adds a dedicated look at hallucination detection.
Reference-based metrics compare a generated summary against human-written gold summaries, which is straightforward but expensive and can penalize valid paraphrases. Context-based metrics instead check the summary against the source document, verifying that claims are grounded in the input. Preference-based metrics rely on human judgments or model rankings to capture what readers actually find useful, though they require careful calibration.
For hallucination detection, the article highlights self-consistency—generating multiple summaries and measuring agreement—as a lightweight signal, alongside entailment-based methods that test whether the summary is logically supported by the source. The takeaway is that no single metric is sufficient; a combination of approaches is needed to balance faithfulness, quality, and cost.