Context compression is a common technique for improving long-context efficiency, but it can introduce faithfulness problems through an information bottleneck. The authors of a new arXiv paper argue that current evaluation methods are insufficient for detecting these issues.
Existing evaluations rely on question-answering metrics, yet a reader can still answer correctly from an unfaithful compressed context. This means QA accuracy alone does not guarantee that the model preserved the original evidence.
To address this, the paper introduces KEEP-Bench, a benchmark designed to evaluate evidence preservation in context compression. The abstract does not detail the benchmark's methodology or results, but its stated goal is to provide a more direct test of faithfulness than QA metrics alone.