Training large language models over extended periods frequently encounters unexpected difficulties. Runs that seemed stable can abruptly diverge, generating invalid outputs such as NaN (Not a Number) losses or exhibiting sudden, sharp increases in the loss function. These events stop progress and consume substantial computational resources.

This chapter concentrates on addressing these training challenges. You will learn to recognize common indicators of instability by monitoring key metrics like loss values and gradient norms ($||\nabla L||$). We will cover methods for diagnosing the underlying causes of issues such as loss spikes and numerical precision errors, especially when using mixed-precision formats ($FP16$ or $BF16$). We will also revisit stabilization methods, including gradient clipping and appropriate learning rate adjustments, and consider how architectural decisions impact training steadiness. By the end of this chapter, you will be better equipped to anticipate, diagnose, and resolve instabilities during large-scale training runs.