Training large language models is a resource-intensive process that often spans days or even weeks on large compute clusters. Given this duration, the probability of encountering hardware failures, software glitches, or unexpected interruptions becomes significant. Without a mechanism to save progress, such interruptions could force training to restart from scratch, wasting substantial compute resources and time.

This chapter addresses the practical necessity of fault tolerance in large-scale training through checkpointing. We will cover the methods required to periodically save the complete state of a training job. You will learn:

• What components constitute the training state (model weights, optimizer states, scheduler states, data loader progress).
• How to implement checkpointing effectively in distributed training environments, ensuring consistency across multiple workers.
• Different strategies for saving checkpoints (synchronous vs. asynchronous) and their associated trade-offs.
• Considerations for determining checkpoint frequency and managing the storage requirements for potentially large checkpoint files.
• The procedures for correctly resuming a training job from a saved checkpoint, including restoring the model, optimizer, and data iteration state.