Long-output inference in large language models is constrained by the key-value (KV) cache, which stores intermediate states and grows with output length. This creates memory and bandwidth bottlenecks that slow generation and limit practicality.

Quantizing the KV cache to lower bit widths reduces these costs, but aggressive quantization, such as 2-bit, tends to degrade model accuracy. The new paper, PN-Mix, introduces a pair-norm guided mixed-precision approach specifically for key cache quantization, aiming to preserve accuracy while achieving efficiency.

By applying mixed precision guided by pair norms, the method targets the most sensitive components of the key cache with higher precision and uses lower bits elsewhere. The authors indicate that this alleviates the accuracy loss seen with uniform low-bit quantization, making long-output inference more feasible.