Language models trained on large web corpora can inadvertently learn or memorize private or unwanted information. A new arXiv preprint proposes Surprise Masking (SM), a straightforward perplexity filtering technique designed to remove such data before training.

The method works by using model likelihoods from a publicly available model to identify and mask surprising tokens. The authors describe SM as a simple form of perplexity filtering, suggesting it can be applied with minimal overhead compared to more complex privacy-preserving approaches.

Because the source is only an abstract, details on implementation and experimental results are not yet available. The proposal highlights a growing concern: as language models become more capable, ensuring training data does not leak sensitive content remains an open challenge.