Fully synthetic pretraining data, generated by symbolic programs, has shown promise in bootstrapping downstream performance on naturalistic tasks such as language modeling. However, the source abstract notes that these generating programs have so far been designed by hand, limiting scalability and exploration.
The new paper introduces PRECOCIAL, which replaces manual program design with an iterative search method. Rather than hand-crafting the symbolic programs, the approach searches over candidate code curriculums during pretraining, aiming to accelerate learning automatically.
Because the abstract is truncated, details of the search procedure and experimental results are not available in the source. The claims here are limited to what the abstract explicitly states: the motivation, the proposed shift from manual to automatic program discovery, and the targeted outcome of accelerated learning.