Post-training quantization (PTQ) for large language models often uses reconstruction loss minimization to keep weights accurate at low bit widths. A new arXiv preprint argues that this standard assumption can backfire: the weights favored by minimizing that loss need not produce a better model.
The paper focuses on weight-only PTQ, where the goal is to preserve model quality at low precision. The authors demonstrate that lower reconstruction loss can be misleading, suggesting that the loss commonly used to guide quantization is not a reliable proxy for final model performance.
To address this, the paper's title points to distributionally robust refinement as the proposed alternative. The abstract does not provide further details of the method, but the finding challenges a core assumption in low-bit quantization research.