The paper introduces RACE (RoPE-Aligned Attention Compression Based on Query-Key Activation Energies), a method for compressing the key-value (KV) cache in transformer models. The KV cache is a major memory bottleneck during inference, and head-dimension compression is one way to reduce its footprint.
However, the abstract notes a key limitation of prior head-dimension compression techniques: they often reconstruct full-dimensional representations during inference, which adds significant decoding FLOPs overhead. RACE is presented as a response to this problem, though the provided excerpt does not detail how it works or what performance it achieves.
Because the abstract is truncated, the article cannot confirm whether RACE successfully eliminates the reconstruction overhead or how it compares to existing methods. The full paper would be needed to evaluate its claims.