Masked language models such as BERT are typically treated as representation learners, even though their pretraining objective is a form of denoising. A new arXiv paper asks whether that hidden capability can be put to work for generation. The authors propose a decoder that leverages BERT’s denoising behavior to produce outputs iteratively, without the dedicated training usually required for diffusion models.
The premise is that the denoising objective already teaches the model to reconstruct coherent text from corrupted input, which is conceptually close to what iterative generation needs. Rather than training a new diffusion model from scratch, the paper explores whether an off-the-shelf MLM can be steered toward conditional generation by exploiting this existing skill.
The abstract does not yet report final results, so the main contribution at this stage is the question and the proposed approach. If the method works, it would offer a cheap way to add generation capabilities to widely available masked language models. If it does not, it would still clarify the boundary between denoising pretraining and true generative modeling.