Large language models are increasingly used to simulate survey respondents, but a new preprint argues that simply matching overall survey averages is insufficient validation. The paper, posted on arXiv, asks whether LLMs can reproduce demographic differences—specifically, within-country age gaps in loneliness.
The authors test LLM outputs against European survey data, focusing on age-related differences. This is a more demanding test than comparing aggregate averages, because a model could match the mean while failing to reflect which demographic groups are more lonely.
The study underscores a methodological concern: if simulated respondents are used for research on social phenomena, they must not only resemble the average respondent but also preserve demographic heterogeneity. The paper therefore highlights the need to validate LLM survey simulations against demographic patterns, not just overall averages.