Representation autoencoders (RAEs), which pair a frozen pretrained self-supervised learning (SSL) encoder with a trained decoder, have recently shown strong results in image generation. In this paper, Miranda, Malard, Richard, Fontaine, and Essid adapt the idea to audio, calling their approach AuRA. They train decoders for two pretrained SSL audio encoders and compare reconstruction quality against the autoencoders used by recent text-to-audio (TTA) systems.
The authors then go a step further: they train rectified flow models in both the SSL spaces and the baseline latent spaces, isolating the autoencoder's effect on generation. Despite differences in raw reconstruction performance, they find that generation in SSL spaces is comparable or superior to generation in the baseline spaces. This suggests that the latent space used for TTA generation need not be the one with the best reconstruction metrics.
The study's main implication is that reconstruction quality alone is an insufficient proxy for generative performance. For TTA researchers, this opens the door to leveraging pretrained SSL representations rather than always training dedicated autoencoders from scratch. The authors note that RAE use in audio remains limited, and this work is an early systematic exploration of that direction.