Post-training quantization (PTQ) compresses the weights of a trained model without full retraining, using limited compute and data. According to a new arXiv abstract, PTQ remains near-lossless down to 4 bits of precision. However, when scalar quantization is pushed to 2 bits, accuracy collapses to chance level, meaning the compressed model performs no better than random guessing.
The paper, titled "Beyond 4 bits: Data-Centric Post-training Quantization and Recovery," investigates this sharp degradation. The title points to a data-centric strategy for recovering performance at very low bit widths, though the abstract excerpt does not detail the specific methods. The authors state they study three areas, but the announcement cuts off before listing them.
This work highlights the gap between 4-bit and 2-bit quantization, and underscores the need for more sophisticated recovery techniques if extreme compression is to become practical.