Target speaker extraction (TSE) aims to isolate one voice from a mixture using a cue that identifies the speaker. Prior systems typically rely on a single cue type—such as an enrollment clip or a text description—and most use mask-prediction models that can introduce audible artifacts and struggle across acoustic conditions. The new PromptSep-Speaker model instead uses a latent diffusion framework that can take any of four cues: speaker temporal order, transcript, enrollment audio, or a natural-language speaker caption.
The authors argue that no single cue is both broadly available and reliably discriminative, so the model is designed to accept whichever cue is at hand. To make full use of each cue, they introduce a frozen WavLM representation with an auxiliary alignment loss that supervises the cross-attention between the query and the mixture, without requiring extra labels. This lets the model learn semantic alignment across modalities.
Across four evaluation sets and a listening study, PromptSep-Speaker achieved state-of-the-art performance compared with TSE and universal source separation baselines, improving both separation accuracy and perceived quality. The results suggest that generative modeling and multimodal prompting together offer a more flexible and robust path for isolating target speakers in real-world conditions.