Blending Synthetic Text with Real-World Data

While using purely synthetic data for pretraining is an option, especially in data-scarce scenarios, a more common and often more effective strategy is to blend synthetic text with existing real-world data. This hybrid approach allows you to leverage the strengths of both data types: the breadth and authenticity of real-world data, and the targeted, controllable nature of synthetic data. The goal is to enrich your pretraining corpus, $V_{data}$, in ways that improve the final model's capabilities.

Why Blend? The Rationale for Hybrid Corpora

Combining synthetic and real data for pretraining isn't just about increasing the sheer volume of text. It's a strategic move to enhance the dataset's quality and utility. Here are some primary motivations:

• Augmenting Scarce Real Data: If your real-world data is limited, high-quality synthetic data can significantly expand your pretraining corpus, providing the model with more examples to learn from.
• Enhancing Diversity: Real-world datasets, even large ones, can have inherent biases or lack coverage in specific topics or styles. Synthetic data can be generated to fill these gaps, introducing more linguistic variety, covering underrepresented domains, or providing examples of rare phenomena.
• Targeted Knowledge Injection: You can create synthetic data focused on specific knowledge areas (e.g., a particular scientific field, a new coding language) or desired skills (e.g., step-by-step reasoning) that might be thinly represented in your general real-world corpus.
• Balancing Data Distributions: If certain categories or styles are overrepresented in your real data, you can use synthetic data to boost the representation of less common but important categories, leading to a more balanced pretraining diet for your LLM.
• Controlled Introduction of New Formats: As mentioned in the chapter introduction, pretraining can benefit from instruction-style data. Synthetic generation is an excellent way to create such data and blend it into a corpus primarily composed of descriptive text.

However, blending is not without its considerations. The quality of synthetic data is critical. Adding large amounts of low-quality or misaligned synthetic data can degrade model performance by introducing noise or conflicting signals.

Determining the Mix: Ratios of Synthetic to Real Data

One of the first practical questions you'll face is: how much synthetic data should you add? There's no universal formula, and the optimal ratio of synthetic to real data, let's call it $R_{S:R}$, depends on several factors:

• Quality of Synthetic Data: Higher quality, more realistic, and diverse synthetic data can generally be used in larger proportions. If your synthetic data is noisy or has noticeable artifacts, use it more sparingly.
• Quantity of Real Data: If you have a vast, diverse real-world corpus, you might only need a small percentage of synthetic data for targeted improvements. If real data is scarce, synthetic data will necessarily form a larger portion.
• Pretraining Goals: If you're aiming to imbue the model with very specific knowledge or skills primarily covered by synthetic data, its proportion might be higher. For general pretraining augmentation, a smaller ratio might suffice.
• Risk Tolerance for "Distributional Shift": Adding too much synthetic data, especially if its characteristics (style, vocabulary, factual basis) differ significantly from real data, can shift the overall data distribution. This might be desirable if you're intentionally trying to adapt the model, but it can also lead to the model underperforming on tasks aligned with the original real-data distribution.

A Common Starting Point: Many practitioners start by augmenting their real-world corpus with a relatively small percentage of synthetic data, perhaps 5% to 20% of the total final volume. For example, if you have 1 Terabyte (TB) of real-world text and aim for a 10% synthetic augmentation, you would add approximately 100 Gigabytes (GB) of synthetic data.

It's often wise to experiment. You might train smaller models or for fewer steps with different $R_{S:R}$ ratios and evaluate their performance on a relevant validation set or a suite of probe tasks to find a sweet spot.

Techniques for Combining Data Streams

Once you've decided on a rough ratio, how do you actually mix the data? Here are a few common approaches:

1. Simple Concatenation: The most straightforward method is to simply append your synthetic dataset to your real-world dataset. If your datasets are stored as collections of files, this might involve adding the synthetic files to the same directory or list used by your data loader.

   • Pros: Easy to implement.
   • Cons: If there are significant differences in size or characteristics, one dataset might dominate certain stages of training if the data isn't thoroughly shuffled. For very large datasets, ensuring effective shuffling can be a challenge in itself.

2. Interleaving or Stratified Mixing: A more controlled approach involves interleaving data from both sources. This can be done at various granularities:

   • File-level interleaving: Alternating between files from real and synthetic sources.
   • Batch-level interleaving: Constructing training batches by drawing samples from both real and synthetic datasets according to the desired ratio. For instance, if you aim for 20% synthetic data, each batch could (on average) contain 20% synthetic examples and 80% real examples.
   • Instance-level shuffling: Pooling all data and then performing a global shuffle. This is ideal for ensuring randomness but can be computationally intensive for terabyte-scale datasets.

   The diagram below illustrates the difference between simple concatenation and an interleaving strategy.

   Comparison of concatenation, where datasets are simply combined, versus interleaving, where data from different sources are mixed more granularly during the creation of the training stream.

Batch-level interleaving is often a good compromise, offering better mixing than simple concatenation without the full overhead of a global shuffle on massive datasets. Here's a Python snippet illustrating how you might create batches with a target synthetic proportion:

```python
import random

# Assume these are large lists or iterators for your actual data
real_data_pool = ["Real example 1", "Real example 2", ...]
synthetic_data_pool = ["Synthetic example A", "Synthetic example B", ...]

# Target proportion of synthetic data in each batch
synthetic_target_proportion = 0.2 # 20% synthetic

def create_blended_batch(batch_size):
    batch = []
    for _ in range(batch_size):
        # Decide whether to pick from synthetic or real pool
        if random.random() < synthetic_target_proportion:
            # Prefer synthetic_data_pool if it's not empty
            if synthetic_data_pool:
                batch.append(random.choice(synthetic_data_pool))
            elif real_data_pool: # Fallback to real if synthetic is exhausted (or not desired proportion-wise)
                batch.append(random.choice(real_data_pool))
        else:
            # Prefer real_data_pool if it's not empty
            if real_data_pool:
                batch.append(random.choice(real_data_pool))
            elif synthetic_data_pool: # Fallback to synthetic if real is exhausted
                batch.append(random.choice(synthetic_data_pool))
    
    # In a real scenario, ensure pools are not empty or handle StopIteration
    # Also, true sampling would involve not picking the same item repeatedly without replacement
    # or using proper data loaders.
    return batch

# Example usage:
# my_batch = create_blended_batch(32)
# print(my_batch)
```
This snippet is illustrative. Real-world data loading pipelines (e.g., using Hugging Face `datasets`, PyTorch `DataLoader`, or TensorFlow `tf.data`) offer more efficient ways to handle large datasets, shuffling, and batching. You would typically configure these loaders to sample from your combined or separate datasets according to your chosen strategy.

3. Up-sampling and Down-sampling Considerations: Blending can also be an opportunity to address imbalances. * Up-sampling with Synthetic Data: If your real-world data has underrepresented categories (e.g., specific genres of text, technical documentation), you can generate synthetic data for these categories and add it, effectively up-sampling them. * Strategic Down-sampling: While generally you want more data, if a particular segment of your real-world data is overwhelmingly dominant and of lower utility, you might consider slightly down-sampling it if high-quality synthetic data can provide better diversity or cover more important areas. This is less common in pretraining where sheer volume is often beneficial, but can be a consideration.

Quality and Coherence in Blended Datasets

When blending datasets, especially from disparate sources like general web text and highly specific synthetic instructions, maintaining overall quality and coherence is important:

• Consistency: Ensure your synthetic data doesn't introduce stylistic elements or factual information that directly contradicts or undermines the real data in a detrimental way. Some controlled contradiction might be useful for teaching robustness, but widespread inconsistencies can confuse the model.
• Domain Alignment: If you're adding domain-specific synthetic data (e.g., medical texts) to a general corpus (e.g., web crawl), be mindful of how this might skew the model's general knowledge. The blending ratio plays a role here.
• Preprocessing Uniformity: Apply consistent text cleaning, tokenization, and formatting steps to both real and synthetic data before blending. Discrepancies in preprocessing can act as unintended signals to the model.
• Iterative Filtering of Synthetic Data: Before blending, rigorously filter your synthetic data. Techniques for this are covered in Chapter 5 ("Advanced Approaches and Data Refinement") and Chapter 6 ("Evaluating Synthetic Data and Addressing Operational Challenges"). A common mistake is to blend raw, unfiltered synthetic output, which can introduce significant noise.

Evaluating the Impact of Blending

The true test of your blending strategy comes from its impact on the pretraining process and the resulting model's performance. Essential areas to monitor include:

• Pretraining Loss Curves: Observe the training and validation loss. Does the blended dataset lead to smoother convergence? Are there unexpected spikes or plateaus?
• Perplexity on Held-out Sets: Measure perplexity on diverse held-out sets, including those representing real-world distributions and those specific to the domains covered by your synthetic data.
• Downstream Task Performance: This is often the most definitive measure. Evaluate the pretrained model on a suite of downstream tasks relevant to your goals. Does the model pretrained on the blended corpus perform better, worse, or differently than one pretrained only on real data (if such a baseline exists)?
• Probing for Specific Knowledge/Skills: If your synthetic data was designed to teach specific things (e.g., a new API, a certain reasoning pattern), develop probes or targeted evaluations to see if the model has acquired these capabilities.

Blending synthetic and real-world data is an art as much as a science. It requires careful consideration of your goals, the nature of your available data, and iterative experimentation. By thoughtfully combining these resources, you can create richer, more effective pretraining corpora that push the boundaries of what your LLMs can achieve. The next sections will get deeper into generating specific types of synthetic data, including instruction-formatted content, that you can then blend using these strategies.