Long chain-of-thought decoding is memory-bound: as a model spends more test-time compute generating reasoning tokens, the KV cache grows linearly. That makes memory a key constraint on how much reasoning a model can perform.
The obvious fix—evicting cache entries—doesn't solve the problem. The abstract notes that a model never trained on a cache with missing entries will not behave correctly after eviction. So the cache must be compressed, not truncated.
AbridgeKV addresses this by compressing long reasoning into the KV cache of learned pause tokens. Rather than discarding information, the method stores a compressed representation in the cache associated with special pause tokens. The available abstract is short, but the direction is clear: make the cache itself the place where reasoning is compactly held.