Quantizing large language models to very low bit widths is an attractive way to reduce deployment costs, but the trade-off in accuracy becomes severe below 3 bits. Scalar quantization, the usual approach, tends to break down in this regime, losing too much model quality to be practical.
SAGE, described in a new arXiv preprint, instead explores vector quantization for 2-bit weight-only compression. The abstract notes that vector quantization is a promising alternative, and the paper's title frames the work as pushing such models toward real-world usability.
Because the abstract is brief, the specific mechanisms and evaluation results are not detailed here. The significance lies in addressing the known accuracy cliff at 2 bits, potentially making extremely compressed LLMs more viable for deployment.