Training large language models presents unique optimization challenges due to their immense scale and computational requirements. Standard optimization techniques often need adaptation for effective LLM training. This chapter focuses on the optimization algorithms and strategies commonly used.

We will begin with a brief review of gradient descent methods before concentrating on adaptive optimizers like Adam and AdamW, explaining the rationale behind decoupled weight decay. You will learn to implement common learning rate scheduling strategies involving warmup and decay phases (e.g., $\eta_{t} = \text{schedule}(\text{step})$). We will also cover gradient clipping, a technique often used to prevent exploding gradients and improve training stability, typically by rescaling gradients whose norm exceeds a threshold $c$: $\mathbf{g} \leftarrow \frac{c}{\|\mathbf{g}\|} \mathbf{g} \text{ if } \|\mathbf{g}\| > c$ Finally, we'll discuss practical guidance for selecting key optimizer hyperparameters, such as the learning rate $\eta$, Adam's momentum terms ($\beta_1, \beta_2$), the numerical stability term $\epsilon$, and the weight decay coefficient $\lambda$.