Long-context large language model inference is important for modern applications, but it is held back by peak memory usage during the prompt prefill phase. Existing KV cache compression techniques have focused on cutting memory during decoding, yet the prefill stage remains a bottleneck.
TideKV, a new method described on arXiv, takes a different approach. It uses layer-wise chunked prefill to delay key-value eviction, which is designed to reduce peak memory during prefill rather than only during decoding. The method is aimed at making long-context inference more memory-efficient.
The abstract notes that existing KV cache compression reduces decoding memory, but it suggests that this is not enough for the prefill phase. TideKV appears to address that gap, though the full details of the method and its evaluation are not included in the abstract.