A new paper on arXiv describes the pretraining data mixture of large language models as their 'digital DNA' — the underlying composition that shapes what a model can do and where it goes wrong. The authors argue that this mixture is central to model behavior, yet it is rarely disclosed by the organizations that build the models.
The paper, titled 'LLMSurgeon: Diagnosing Data Mixture of Large Language Models,' positions post-hoc auditing of data mixtures as a key challenge. Because the composition is not published, outside researchers have limited ability to trace a model's strengths and failures back to its training data. The authors present LLMSurgeon as a diagnostic approach aimed at this problem.
This summary is based on a single preprint, and the abstract provided is truncated, so the specific mechanics of LLMSurgeon are not detailed here. The core claim stands: data mixture is a major determinant of LLM behavior, and better tools are needed to audit it after the fact.