Mixture-of-experts (MoE) language models activate only a few experts per token, yet all expert weights must be stored in memory. Mixed-precision quantization reduces this storage cost by assigning higher bit-widths to sensitive experts and lower bit-widths to others. A new preprint on arXiv, titled "Beyond Expert Rankings: Contrast Calibration for Mixed-Precision MoE Quantization," addresses how to decide which experts are sensitive.
The authors argue that existing methods rely on expert rankings, which may not capture the full picture. Instead, they propose a contrast calibration approach for setting mixed-precision bit-widths. The abstract does not provide experimental details, so the reported gains and comparisons are not yet available from the source.
As a preprint, the work has not been peer-reviewed. The key contribution, according to the abstract, is a calibration strategy that goes beyond simple ranking to better allocate bits across experts in MoE models.