A new study on arXiv tackles the persistent challenge of abstractive summarization across languages. The authors note that while large language models can generate fluent summaries, their factual consistency may not always match, and this gap tends to widen in low-resource settings.
To address this, the study presents a standardized zero-shot benchmark designed to evaluate models without task-specific fine-tuning. The benchmark provides a common framework for comparing performance on English and Bengali summarization, offering a clearer view of how models handle both languages under the same conditions.
The work emphasizes that linguistic quality and factual consistency are separate dimensions that can diverge, particularly when resources are scarce. This suggests that future evaluations should assess both aspects explicitly rather than relying on a single overall score.