Long contexts in language models lead to large key-value (KV) caches, which increase memory use and attention costs during generation. Compressing these caches is challenging because the compressed representation must still support answering questions that were not known at compression time.
The arXiv paper introduces Latent Compaction, a method that learns to compress KV caches. The abstract states that the approach is designed to preserve information for unseen questions, though the full details of the learning procedure are not included in the abstract. The title indicates that test-time training is involved.
This work addresses a practical bottleneck for deploying models with very long contexts, where cache size can dominate memory. By reducing cache footprint while retaining query-relevant information, such methods could lower inference costs.