So far, our focus has been on value-based reinforcement learning methods. We learned to estimate the value of states ($V(s)$) or state-action pairs ($Q(s,a)$) and derived policies based on these values. This chapter introduces a distinct approach: policy gradient methods.

With policy gradient methods, we learn a parameterized policy, denoted $\pi_\theta(a|s)$, directly. Instead of estimating value functions first, we aim to optimize the policy parameters $\theta$ to maximize the expected return. This approach is particularly useful in environments with continuous action spaces or when we want to learn stochastic policies.

In this chapter, you will learn:

• The fundamental concept of directly parameterizing and optimizing a policy.
• The core idea behind the Policy Gradient Theorem, which provides the theoretical basis for these methods.
• The REINFORCE algorithm, a foundational Monte Carlo policy gradient technique.
• How using baselines can help reduce the variance inherent in policy gradient estimates.
• A brief overview of Actor-Critic methods, which combine aspects of both value-based and policy-based learning.
• The trade-offs between policy-based and value-based methods.

We will cover the algorithms conceptually and guide you through implementing a basic REINFORCE agent.