Latent diffusion models depend on the quality of the VAE latent space, yet not all latents are equally easy to denoise. The authors of this preprint derive distribution-free bounds on the maximum achievable reduction in pixel-space denoising loss. Under a power-law power spectral density (PSD), these bounds show that sufficiently separated frequencies are ordered, with larger achievable reductions at lower frequencies.

From this analysis, they propose the Spectrum Matching Hypothesis: good latents should have a flattened power-law PSD (Encoding Spectrum Matching) and preserve frequency-to-frequency semantic correspondence through the decoder (Decoding Spectrum Matching). In practice, ESM matches the PSD between images and latents, while DSM uses shared spectral masking with frequency-aligned reconstruction. The authors argue this gives a unified spectral interpretation of earlier methods such as VA-VAE and EQ-VAE, and clarifies why some latents appear over-noisy or over-smoothed.

Experiments on CelebA and ImageNet suggest Spectrum Matching yields superior diffusion generation and outperforms prior approaches. The paper also extends the spectral view to representation alignment (REPA), showing that the directional spectral energy of the target representation is crucial, and proposes a DoG-based method to further improve REPA. Code is released in the multiVAE repository.