Quantizing large language models to ultra-low bit widths can make them cheaper and faster to deploy, but it risks changing their predictions. A new arXiv paper presents FidelityQ, a method designed to preserve predictive fidelity under such extreme compression.
The abstract notes that ultra-low-bit scalar quantization offers a regular arithmetic structure, which is beneficial for efficient deployment. However, it also highlights the difficulty of maintaining predictive behavior when the number of reconstruction possibilities is very small, as is the case at extreme precision.
FidelityQ is proposed as a response to this challenge, though the abstract does not elaborate on the specific mechanism. The paper's focus is on keeping the quantized model's predictions faithful to the original, despite the aggressive reduction in bit width.