Direct Policy Parameterization

While powerful, value-based methods like DQN face certain limitations. These methods primarily focus on learning accurate action-value estimates, $Q(s, a)$, and then derive a policy (often $\epsilon$-greedy) from these values. This indirect approach can become challenging in environments with expansive, continuous action spaces, where finding the maximum Q-value involves an optimization step at each decision point. Furthermore, sometimes the optimal policy itself is inherently stochastic, meaning the agent should choose actions randomly according to a specific probability distribution, which deterministic policies derived from Q-values struggle to represent naturally.

Policy gradient methods offer a different path. Instead of learning values first, we directly learn the policy itself. We introduce a function, the policy function (or simply, the policy), that maps states to actions (or probabilities over actions). This function has its own set of parameters, which we'll denote collectively as $\theta$. Our goal is to adjust these parameters $\theta$ to produce the best possible behavior.

We represent this parameterized policy as $\pi(a|s; \theta)$. This indicates the probability (or probability density for continuous actions) of selecting action $a$ when in state $s$, given the policy parameters $\theta$.

Representing the Policy

How do we actually represent this function $\pi(a|s; \theta)$? Just like we used neural networks to approximate Q-values in DQN, we commonly use neural networks to represent the policy.

• Input: The network takes the current state $s$ as input.
• Network Body: A series of hidden layers process the state information.
• Output: The network's output depends on the type of action space:
  • Discrete Actions: For environments with a finite set of actions (like CartPole's left/right moves), the network typically outputs a probability distribution over these actions. A common choice for the final layer is a softmax activation function, which outputs positive values that sum to 1, representing the probability of taking each action. For example, if there are two actions, the network might output $[0.7, 0.3]$, meaning a 70% chance of taking action 1 and a 30% chance of taking action 2.
  • Continuous Actions: For environments where actions are continuous values (like setting the angle of a robot arm or the acceleration of a car), the network outputs parameters defining a probability distribution. For instance, it might output the mean ($\mu$) and standard deviation ($\sigma$) of a Gaussian (Normal) distribution. To select an action, we then sample from this distribution, $a \sim \mathcal{N}(\mu(s; \theta), \sigma(s; \theta)^2)$. This allows the agent to explore actions smoothly within a range.

The weights and biases of this neural network constitute the parameters $\theta$. Learning the policy means finding the values of $\theta$ that lead to the agent achieving the highest possible cumulative reward.

A diagram illustrating how a policy network takes a state as input and outputs parameters defining the action selection strategy. The parameters $\theta$ are the weights and biases within the network.

The Objective: Maximizing Return

By parameterizing the policy directly, we shift the optimization problem. We are no longer trying to accurately estimate Q-values. Instead, we are searching for the policy parameters $\theta$ that maximize an objective function $J(\theta)$, which typically represents the expected total discounted return obtained by following policy $\pi(a|s; \theta)$.

$$J(\theta) = \mathbb{E}_{\tau \sim \pi_\theta} \left[ \sum_{t=0}^{T} \gamma^t R(s_t, a_t) \right]$$

Here, $\tau$ represents a trajectory (a sequence of states and actions) generated by following the policy $\pi_\theta$, $\gamma$ is the discount factor, and $R(s_t, a_t)$ is the reward received. Our goal is to find the $\theta$ that makes $J(\theta)$ as large as possible.

This direct parameterization offers several advantages:

1. Natural Handling of Continuous Actions: Sampling from distributions like Gaussians is straightforward.
2. Learning Stochastic Policies: The policy can explicitly output probabilities, allowing it to learn optimal stochastic behaviors where needed.
3. Simpler Action Selection: Once the policy is learned, selecting an action simply involves feeding the state to the network and sampling from the output distribution (or taking the most likely action), avoiding the max operation required in Q-learning.

Now that we understand how to represent the policy using parameters $\theta$, the next step is to determine how to adjust these parameters effectively to improve the agent's performance. This leads us to the core mathematical foundation of these methods: the Policy Gradient Theorem.