Building upon first-order gradient methods, this chapter examines optimization techniques that utilize second-order derivative information. These methods consider the curvature of the loss function, often leading to faster convergence compared to gradient descent, particularly when close to a minimum.

We will begin with Newton's method, exploring its theoretical basis which involves approximating the objective function locally with a quadratic model:

$$f(x_k + p) \approx f(x_k) + \nabla f(x_k)^T p + \frac{1}{2} p^T \nabla^2 f(x_k) p$$

The core component here is the Hessian matrix $\nabla^2 f(x_k)$. We will discuss its properties and the computational challenges associated with its calculation and inversion, especially for high-dimensional machine learning problems.

To address these challenges, we will focus on Quasi-Newton methods, which approximate the Hessian or its inverse. You will learn about the popular BFGS algorithm and its memory-efficient adaptation, L-BFGS, widely used in practice. Additionally, we will cover trust-region methods, offering an alternative strategy for managing step sizes using curvature information. The chapter includes a practical section on implementing L-BFGS.