Full-duplex spoken dialogue systems need to listen and speak simultaneously, which requires training data with persistent, speaker-resolved tracks. Such data is hard to obtain at scale. Generative recovery from ordinary recordings is a possible route, but simply separating two voices is insufficient because track assignments can drift over time.
The authors propose a speaker-anchored routing module placed before a frozen diffusion generator. The 1.84M-parameter module takes the separator's coarse streams, matches them to fixed enrollment identities, and orders them, without retraining the generator or altering the reconstructed audio. This addresses the missing link between separation and consistent speaker identity.
On 25-minute remote conversations, the routing reduced wrong-track time from 43.9% to 4.1%, below the 6.2% achieved by matched-enrollment waveform re-identification. The largest improvement occurred when one participant held the floor (17.8% versus 28.6%). Fixed-track activity F1 rose from 0.46 to 0.84 while per-chunk reconstruction quality remained unchanged.
The router also transferred to unseen face-to-face and telephone recordings. A margin-gated verification step brought wrong-track time down to 0.9-1.7% while embedding only 20-36% of the output, compared to full re-identification which embeds all of it. The authors argue that recovered voices can thus become the participant-consistent tracks needed for full-duplex dialogue modeling.