Learning Policies Directly

In the preceding chapters, we explored methods centered around learning the value of being in a particular state or taking a specific action in a state. Algorithms like Q-learning and SARSA first estimate value functions (like $Q(s, a)$) and then use these estimates to determine the best actions, often by selecting the action with the highest estimated value. This approach is powerful but faces challenges, especially when dealing with continuous action spaces or when the optimal behavior itself is inherently random (stochastic).

Policy gradient methods offer an alternative perspective. Instead of learning value functions, we directly learn the policy itself. We define the policy as a parameterized function, let's denote it as $\pi_\theta(a|s)$, which outputs the probability of taking action $a$ in state $s$. The parameters of this function are represented by $\theta$. For example, $\theta$ could be the weights and biases of a neural network.

The core idea is to adjust the parameters $\theta$ directly to improve the quality of the policy. But what does "improve" mean? In reinforcement learning, our objective is typically to maximize the expected total discounted return. We can define a performance measure, often denoted as $J(\theta)$, which represents this expected return achieved by following policy $\pi_\theta$. For episodic tasks, this is often the value of the starting state:

$$J(\theta) = V^{\pi_\theta}(s_0) = E_{\pi_\theta} \left[ \sum_{t=0}^{T} \gamma^t R_{t+1} | S_0 = s_0 \right]$$

Our goal becomes finding the parameters $\theta$ that maximize this performance measure $J(\theta)$.

Why Learn Policies Directly?

Parameterizing the policy directly offers several advantages:

1. Continuous Action Spaces: Value-based methods like Q-learning struggle with continuous actions. Finding the action $a$ that maximizes $Q(s, a)$ requires an optimization procedure at each step if the action space is continuous. Policy gradient methods handle this more naturally. The parameterized policy $\pi_\theta(a|s)$ can be designed to directly output the parameters of a probability distribution over actions (e.g., the mean and standard deviation of a Gaussian distribution for a continuous action) or the continuous action itself.
2. Stochastic Policies: In some scenarios, the optimal policy is stochastic. Consider a game like Rock-Paper-Scissors against an opponent who can exploit deterministic strategies. A random (stochastic) policy is optimal. Value-based methods often implicitly learn deterministic policies (e.g., by always choosing the action with the maximum Q-value). Policy gradient methods can learn explicitly stochastic policies by outputting probabilities $\pi_\theta(a|s)$.
3. Simpler Representation: Sometimes, the policy function is simpler to approximate than the value function. A direct parameterization might require fewer parameters or a less complex function approximator compared to representing the potentially intricate shape of a value function.

Optimization via Gradient Ascent

Since we have a parameterized function $\pi_\theta$ and an objective function $J(\theta)$ we want to maximize, this looks like an optimization problem. We can use techniques similar to those used in supervised learning, but instead of gradient descent (to minimize loss), we use gradient ascent (to maximize performance).

The basic update rule for the policy parameters $\theta$ is:

$$\theta_{k+1} = \theta_k + \alpha \nabla_\theta J(\theta_k)$$

Here:

• $\theta_k$ represents the policy parameters at iteration $k$.
• $\alpha$ is a positive step-size parameter, often called the learning rate.
• $\nabla_\theta J(\theta_k)$ is the gradient of the performance measure with respect to the policy parameters $\theta$. This gradient vector points in the direction of the steepest increase in $J(\theta)$.

By repeatedly calculating the gradient and taking steps in that direction, we iteratively improve the policy parameters $\theta$ to achieve higher expected returns.

Parameterizing the Policy

How do we actually represent $\pi_\theta(a|s)$? The choice depends on the nature of the action space.

• Discrete Actions: A common approach is to use a function approximator (like a neural network) that takes the state $s$ as input and outputs a score or preference $h(s, a, \theta)$ for each possible action $a$. These preferences can then be converted into probabilities using a softmax function:

  $$\pi_\theta(a|s) = \frac{e^{h(s, a, \theta)}}{\sum_{a'} e^{h(s, a', \theta)}}$$

  The parameters $\theta$ are the parameters of the function approximator (e.g., the network weights).

• Continuous Actions: A popular method is to have the function approximator output the parameters of a probability distribution. For instance, for a single continuous action, the network might output the mean $\mu_\theta(s)$ and standard deviation $\sigma_\theta(s)$ of a Gaussian distribution. Actions are then sampled from this distribution:

  $$a \sim \mathcal{N}(\mu_\theta(s), \sigma_\theta(s)^2)$$

  The policy probability density is then given by the Gaussian probability density function (PDF). The parameters $\theta$ are again the parameters of the network determining $\mu$ and $\sigma$.

The main challenge, and the focus of much of this chapter, lies in estimating the gradient $\nabla_\theta J(\theta)$. Unlike supervised learning where we have explicit labels, in RL, the "correctness" of an action is only revealed through the subsequent rewards received over potentially long trajectories. The Policy Gradient Theorem, which we introduce next, provides a theoretical foundation for calculating or estimating this important gradient.