The pretraining phase of Large Language Models demands vast quantities of text data. When real-world data is scarce, insufficient, or lacks specific characteristics, synthetic data offers a viable alternative for building or supplementing pretraining datasets. This chapter examines the application of synthetic data specifically to this foundational stage of LLM development.

You will learn to:

• Understand the relationship between data volume, $V_{data}$, and pretraining effectiveness.
• Construct large-scale synthetic corpora suitable for the pretraining phase.
• Implement strategies for combining synthetic text with existing real-world data.
• Use synthetically generated content for domain-specific or targeted pretraining.
• Generate instruction-formatted data to incorporate into pretraining.
• Evaluate how synthetic data influences the outcomes of pretraining.
• Gain practical experience by assembling a small synthetic pretraining dataset sample.