Synthetic bootstrapped pretraining

We introduce Synthetic Bootstrapped Pretraining (SBP), a language model (LM) pretraining procedure that first learns a model of relations between documents from the pretraining dataset and then leverages it to synthesize a vast new corpus for joint training. While the standard pretraining teaches LMs to learn causal correlations among tokens within a single document, it is not designed to efficiently model the rich, learnable inter-document correlations that can potentially lead to better performance. We validate SBP by designing a compute-matched pretraining setup and pretrain a 3B-parameter and a 6B-parameter model on up to 1T tokens from scratch. We find SBP consistently improves upon a strong repetition baseline and delivers up to 60% of performance improvement attainable by an oracle upper bound with access to 20x more unique data. Qualitative analysis reveals that the synthesized documents go beyond mere paraphrases – SBP first abstracts a core concept from the seed material and then crafts a new narration on top of it. Besides strong empirical performance, SBP admits a natural Bayesian interpretation: the synthesizer implicitly learns to abstract the latent concepts shared between related documents.

💡 Research Summary

The paper introduces “Synthetic Bootstrapped Pretraining” (SBP), a novel pretraining methodology designed to overcome the limitations of standard language model (LM) training. Traditional pretraining focuses on learning causal correlations between tokens within a single document (intra-document correlation). However, this approach fails to capture the rich, learnable inter-document correlations that exist across a dataset. SBP addresses this by first learning a model of relations between documents from an existing dataset and then utilizing this model to synthesize a vast, new corpus for joint training.

The core technical innovation of SBP lies in its ability to move beyond simple data augmentation or paraphrasing. The authors demonstrate that the SBP synthesizer does not merely rewrite existing text; instead, it performs a sophisticated process of abstraction and narration. The synthesizer first abstracts core concepts from the seed material and then crafts entirely new narrations based on those abstracted concepts. This process allows the model to capture higher-order semantic structures that are often missed in standard training regimes. From a theoretical perspective, the authors provide a natural Bayesian interpretation, suggesting that the synthesizer implicitly learns to identify and model the latent concepts shared between related documents.

To validate the effectiveness of SBP, the researchers conducted large-scale experiments using 3B and 6B parameter models, trained on up to 1 trillion tokens from scratch. The experimental setup was carefully designed to be compute-matched, ensuring a fair comparison with existing baselines. The results were highly significant: SBP consistently outperformed strong repetition-based baselines. Most impressively, the researchers found that SBP can achieve up to 60% of the performance gains attainable by an “oracle” upper bound—a hypothetical scenario where the model has direct access to 20x more unique, non-synthetic data. This indicates that SBP provides a highly efficient way to scale model intelligence without the immediate need for massive amounts of new, human-written data.

In summary, SBP represents a paradigm shift in how we approach the scaling laws of language models. By shifting the focus from token-level patterns to inter-document relational modeling, SBP demonstrates that synthetic data, when generated through concept-driven abstraction, can serve as a powerful engine for bootstrapping model performance. This research opens new avenues for using generative models to augment the quality and depth of pretraining datasets, potentially breaking the bottleneck of human-generated data scarcity in the era of massive-scale AI development.