A new arXiv paper takes up a core question for AI alignment: how do language models generalize from their training data? The authors argue that while models reliably learn the distribution they are trained on, the behaviors that matter for real-world deployment are those that hold up in new, out-of-distribution contexts. This gap between training and deployment is a central challenge for ensuring models behave as intended.

The paper frames the problem in terms of "propensities"—the tendencies or dispositions a model develops during training. The goal is to predict which of these propensities will generalize out of distribution, rather than merely reflecting the statistical regularities of the training set. If such predictions can be made reliably, it would allow researchers to anticipate alignment failures before a model is deployed.

Because the abstract is truncated, the specific methods or results are not available in the source. However, the framing itself is significant: it shifts attention from measuring in-distribution performance to forecasting how a model's learned behaviors will transfer to unfamiliar situations, which is precisely where alignment risks often emerge.