Previous chapters focused on methods like Deep Q-Networks (DQN) which learn the value $Q(s, a)$ of taking actions in states. While effective, these value-based approaches can struggle in certain situations, such as environments with continuous action spaces or when a stochastic policy is inherently required.

This chapter introduces Policy Gradient methods, a fundamentally different strategy. Here, we directly learn a parameterized policy $\pi(a|s; \theta)$ that selects actions, without relying on an intermediate value function estimate for action selection. We will begin by discussing the limitations of value-based methods that motivate this alternative approach.

We will then examine the core idea behind policy gradients: adjusting the policy parameters $\theta$ to maximize expected return. This involves understanding the Policy Gradient Theorem, the theoretical basis for these methods. You will learn to implement the REINFORCE algorithm, a foundational Monte Carlo policy gradient technique. We will also address the challenge of high variance often encountered with REINFORCE and introduce the concept of using baselines to improve stability and convergence speed. The chapter concludes with a practical implementation exercise for the REINFORCE algorithm.