Zero-shot text-to-speech models can clone a voice from a few seconds of audio, but the largest versions rely on billion-parameter diffusion transformers that are hard to run on-device. Post-training quantization (PTQ) can shrink such models without retraining, yet quantizing to 4 bits has a specific failure mode: intelligibility stays close to full precision while speaker timbre drifts. The authors of TTS-Q set out to find where that damage lives and whether it can be repaired.
By running a 4-bit weight-and-activation (W4A4) PTQ study at 1B and 3.5B scales, they found the timbre loss comes from both weight and activation quantization. Only the activation side, however, has a timestep structure that a precision schedule can exploit. Their method, TTS-Q, selects hard-to-quantize items from task-matched calibration candidates and applies a budget-balanced mixed-precision schedule that gives higher precision to late-step feed-forward activations.
Across both model scales, TTS-Q recovers speaker timbre best while keeping intelligibility intact. The paper also reports a calibrated W4A4 reference that reduces peak GPU memory, offering a practical option for resource-constrained TTS deployment.