Researchers from the new arXiv preprint study how two design choices shape multilingual pre-training when data is scarce. The work is situated in the BabyLM challenge, where models are trained on a relatively small corpus meant to approximate the language input a child receives. The authors specifically examine English, Dutch, and Chinese, looking at both the proportion of each language in the training mix and the sequence in which the languages are encountered.
The paper distinguishes between distribution and order as separate variables. Distribution refers to how much of the training data comes from each language, while order concerns the temporal arrangement of language exposure during pre-training. By comparing different settings, the authors aim to identify whether certain mixes or sequences lead to better learning outcomes under the constrained BabyLM setup.
The abstract is truncated, so the specific comparisons and results are not fully available in the source. Nevertheless, the stated goal is clear: to understand how language exposure, in both amount and sequence, affects multilingual models when data is limited. As this is a single source, there are no conflicting findings to reconcile.