Lipreading is hard because visual cues are ambiguous and degrade with head pose or low resolution. In a paper accepted to NeurIPS 2026, Jing-Xuan Zhang and colleagues introduce FLARE, a framework that learns visual speech representations by borrowing supervision from an audio model.
FLARE uses Whisper's semantically rich representations as a teacher signal, converting them into discrete distillation targets via a random-projection quantizer and pretraining with a masked prediction objective. The architecture has two streams: one processes the lip region for fine articulatory motion, the other processes the full face for broader dynamics. Audio is only used during pretraining to align modalities; a modality dropout strategy encourages the visual side to stand on its own.
On the LRS2, LRS3, and out-of-domain WildVSR benchmarks, FLARE reports word error rates of 11.7%, 13.2%, and 34.3%. The authors' ablations show each component contributes, and joint face-lip modeling improves robustness under degraded resolution and large head pose variation. The authors also find the learned representations are more phoneme-discriminative.
Code and pretrained models are publicly available.