Baselines for Variance Reduction

The REINFORCE algorithm, while foundational for policy gradients, often struggles in practice due to the high variance inherent in its gradient estimates. The policy update relies on the total return $G_t$ observed after taking action $a_t$ in state $s_t$. Because $G_t$ can vary significantly depending on the subsequent stochastic transitions and actions, the resulting gradient estimates $\nabla_\theta \log \pi_\theta(a_t|s_t) G_t$ can be noisy. This noise slows down learning and can prevent the policy from converging reliably to a good solution. Imagine trying to adjust a knob based on wildly fluctuating feedback; it's hard to know which direction is truly better.

Actor-Critic methods offer a compelling solution by introducing a baseline into the policy gradient calculation. The core idea is surprisingly simple: subtract a value from the return $G_t$ that depends only on the state $s_t$, let's call it $b(s_t)$. The modified policy gradient update term becomes:

$$\nabla_\theta \log \pi_\theta(a_t|s_t) (G_t - b(s_t))$$

Why does this work? Crucially, subtracting a state-dependent baseline does not change the expected value of the policy gradient, meaning it doesn't introduce bias into the update direction. We can show this mathematically. The expected value of the term we subtract is:

$$\mathbb{E}_{\pi_\theta} \left[ \nabla_\theta \log \pi_\theta(a_t|s_t) b(s_t) \right] = \sum_s d^{\pi_\theta}(s) \sum_a \pi_\theta(a|s) \nabla_\theta \log \pi_\theta(a|s) b(s)$$

Using the identity $\nabla_\theta \log \pi_\theta(a|s) = \frac{\nabla_\theta \pi_\theta(a|s)}{\pi_\theta(a|s)}$, this becomes:

$$\sum_s d^{\pi_\theta}(s) \sum_a \nabla_\theta \pi_\theta(a|s) b(s) = \sum_s d^{\pi_\theta}(s) b(s) \nabla_\theta \sum_a \pi_\theta(a|s)$$

Since $\sum_a \pi_\theta(a|s) = 1$ for any state $s$, its gradient with respect to $\theta$ is zero: $\nabla_\theta 1 = 0$. Therefore, the expected value of the subtracted term is zero:

$$\mathbb{E}_{\pi_\theta} \left[ \nabla_\theta \log \pi_\theta(a_t|s_t) b(s_t) \right] = 0$$

This confirms that subtracting a state-dependent baseline $b(s_t)$ maintains the unbiasedness of the policy gradient estimate.

How Baselines Reduce Variance

While the expected gradient remains the same, the variance of the gradient estimate can be significantly reduced. Consider the term $(G_t - b(s_t))$. If we choose $b(s_t)$ to be a good estimate of the average return from state $s_t$, then this term represents how much better or worse the actually observed return $G_t$ was compared to the average expectation from that state.

• If $G_t > b(s_t)$: The action $a_t$ led to a better-than-average outcome. The term is positive, reinforcing action $a_t$.
• If $G_t < b(s_t)$: The action $a_t$ led to a worse-than-average outcome. The term is negative, discouraging action $a_t$.

By centering the returns around a state-specific average, the magnitude of the updates is scaled down. Instead of large positive updates based on high absolute returns (which might just come from a generally rewarding state), the updates focus on the relative quality of the action taken in that specific context. This leads to a more stable and often faster learning process.

Sample returns ($G_t$) from a trajectory. Subtracting a baseline (here, the overall average return) centers the values used for policy updates around zero. Positive values correspond to returns better than the baseline, negative values to returns worse than the baseline. This centering helps reduce the update variance.

The State-Value Function as the Optimal Baseline

What is the best choice for $b(s_t)$? While a simple constant baseline (like the average return over an episode) can help somewhat, a much more effective baseline is the state-value function, $V^{\pi_\theta}(s_t)$. This function, by definition, represents the expected return starting from state $s_t$ and following the current policy $\pi_\theta$.

Using $V(s_t)$ as the baseline, the update term becomes:

$$\nabla_\theta \log \pi_\theta(a_t|s_t) (G_t - V(s_t))$$

The term $(G_t - V(s_t))$ is an estimate of the Advantage Function, $A^{\pi_\theta}(s_t, a_t)$. The advantage function measures how much better taking action $a_t$ in state $s_t$ is compared to the average action chosen by the policy $\pi_\theta$ from state $s_t$. Formally:

$$A^{\pi_\theta}(s_t, a_t) = Q^{\pi_\theta}(s_t, a_t) - V^{\pi_\theta}(s_t)$$

where $Q^{\pi_\theta}(s_t, a_t)$ is the action-value function. Since $G_t$ is a Monte Carlo sample estimate of $Q^{\pi_\theta}(s_t, a_t)$, the term $(G_t - V(s_t))$ serves as a sample estimate of the advantage $A^{\pi_\theta}(s_t, a_t)$.

Using the state-value function $V(s_t)$ as the baseline is theoretically shown to be the optimal choice (in the sense of minimizing the variance of the gradient estimate) among all functions that depend only on the state $s_t$.

The Emergence of the Critic

This naturally leads us to the Actor-Critic architecture. We need a way to estimate $V(s_t)$ to use it as a baseline. This is precisely the role of the critic.

• The Actor is responsible for learning and updating the policy parameters $\theta$, typically using gradient ascent with the advantage estimate: $$\theta \leftarrow \theta + \alpha \nabla_\theta \log \pi_\theta(a_t|s_t) A(s_t, a_t)$$ (where $A(s_t, a_t)$ is estimated, e.g., by $G_t - V(s_t)$ or other methods we'll see later).
• The Critic is responsible for learning an estimate of the state-value function $V(s_t; \phi)$ (or sometimes the action-value function $Q(s_t, a_t; \phi)$), parameterized by $\phi$. It learns from the transitions experienced by the actor, often using Temporal Difference (TD) learning methods like TD(0). The critic's output $V(s_t; \phi)$ is then used by the actor as the baseline $b(s_t)$.

Diagram of an Actor-Critic architecture using $V(s)$ as a baseline. The Actor selects actions based on the policy. The Critic evaluates states by learning $V(s)$. The Critic's value estimate $V(s)$ is used as a baseline to compute an Advantage estimate, which in turn provides a lower-variance signal for updating the Actor's policy. Both components learn from interactions with the environment.

In summary, introducing a state-dependent baseline, particularly the state-value function $V(s_t)$, is a powerful technique for reducing the variance of policy gradient estimates without introducing bias. This naturally motivates the Actor-Critic framework, where the critic learns the value function to provide this baseline, and the actor updates the policy using the resulting advantage signal. The following sections will explore specific algorithms like A2C/A3C that implement this idea effectively and introduce further refinements like Generalized Advantage Estimation (GAE).