A new preprint on arXiv proposes ACE-Compression, an ability-centric framework for evaluating compressed large language models. The authors argue that current evaluation practice is incomplete: it typically measures how well a compressed model retains performance on benchmark tasks, then treats that as the full picture.
That approach, they contend, misses a critical fact—compression perturbs the model's underlying abilities, not merely its scores on specific tasks. By focusing only on task performance, standard evaluations can hide or misrepresent the true impact of compression on what the model can actually do.
ACE-Compression is put forward as a corrective: an evaluation method centered on abilities rather than isolated tasks. The preprint is available on arXiv, though the abstract provided does not detail the specific abilities assessed or the experimental results.