Generalized Advantage Estimation (GAE)

Basic policy gradients, such as REINFORCE, suffer from high variance. The Advantage Actor-Critic (A2C) framework addresses this by using a state-value function $V(s)$ as a baseline. In A2C, the policy gradient update depends on the advantage function, $A(s_t, a_t) = Q(s_t, a_t) - V(s_t)$ . The central question then becomes: how do we best estimate this advantage function using the data collected by the agent?

Different estimators for $A(s_t, a_t)$ exist, each with its own trade-offs between bias and variance. Remember, our goal is to get an estimate that leads to stable and efficient policy updates.

Consider the TD error at time $t$ :

\delta_t = r_{t+1} + \gamma V(s_{t+1}) - V(s_t)

This $\delta_t$ is an estimate of the advantage $A(s_t, a_t)$ . Why? Because $r_{t+1} + \gamma V(s_{t+1})$ is an estimate of $Q(s_t, a_t)$ . Using $\delta_t$ directly as the advantage estimate offers low variance because it depends only on the next reward and the value function estimate at the next state. However, it can be biased, especially if the value function estimate $V(s)$ is inaccurate. This is essentially the advantage estimate used in basic one-step Actor-Critic.

Alternatively, we could use the Monte Carlo estimate. We estimate the advantage as the total discounted return $G_t$ minus the baseline $V(s_t)$ :

\hat{A}_t^{MC} = G_t - V(s_t) = \left( \sum_{k=0}^{\infty} \gamma^k r_{t+k+1} \right) - V(s_t)

In practice, we use the return accumulated until the end of an episode (or a fixed horizon). This estimate is unbiased if $V(s_t)$ is the true value function, but it suffers from high variance because it sums potentially many stochastic rewards $r_{t+k+1}$ over the trajectory.

We can also define N-step advantage estimates that blend these two extremes:

\hat{A}_t^{(N)} = \left( \sum_{k=0}^{N-1} \gamma^k r_{t+k+1} + \gamma^N V(s_{t+N}) \right) - V(s_t)

This uses $N$ steps of real rewards and then bootstraps using the value function estimate $V(s_{t+N})$ . Increasing $N$ generally reduces bias but increases variance. This suggests a spectrum of possible estimators.

Generalized Advantage Estimation (GAE)

Generalized Advantage Estimation, or GAE, provides a more sophisticated way to navigate this bias-variance trade-off. It introduces a parameter, $\lambda$ (where $0 \le \lambda \le 1$ ), to explicitly control the weighting between bias and variance. The GAE formula computes the advantage estimate $\hat{A}_t$ as an exponentially weighted average of TD errors $\delta_{t+l}$ over multiple time steps:

\hat{A}_t^{GAE(\gamma, \lambda)} = \sum_{l=0}^{\infty} (\gamma \lambda)^l \delta_{t+l}

Here, $\gamma$ is the standard discount factor, and $\lambda$ is the GAE parameter that tunes the trade-off. Let's examine the boundary cases for $\lambda$ :

Case $\lambda = 0$ : If we set $\lambda = 0$ , the sum collapses to only the first term ( $l=0$ ):
$\hat{A}_t^{GAE(\gamma, 0)} = (\gamma \cdot 0)^0 \delta_t = \delta_t = r_{t+1} + \gamma V(s_{t+1}) - V(s_t)$
This is exactly the one-step TD error estimate. It typically has low variance but can be biased if the value function $V$ is inaccurate.
Case $\lambda = 1$ : If we set $\lambda = 1$ , the formula becomes:
$\hat{A}_t^{GAE(\gamma, 1)} = \sum_{l=0}^{\infty} \gamma^l \delta_{t+l} = \sum_{l=0}^{\infty} \gamma^l (r_{t+l+1} + \gamma V(s_{t+l+1}) - V(s_{t+l}))$
This sum telescopes. If we ignore function approximation errors and assume $V$ is the true value function, this sum converges to:
$\hat{A}_t^{GAE(\gamma, 1)} \approx \left( \sum_{l=0}^{\infty} \gamma^l r_{t+l+1} \right) - V(s_t) = G_t - V(s_t)$
This is essentially the unbiased Monte Carlo advantage estimate, which tends to have high variance.
Intermediate $\lambda$ ( $0 < \lambda < 1$ ): Values of $\lambda$ between 0 and 1 create estimates that interpolate between the TD(0) advantage and the Monte Carlo advantage. A higher $\lambda$ (closer to 1) gives more weight to longer-term reward information, reducing bias at the cost of increased variance. A lower $\lambda$ (closer to 0) relies more heavily on the current value function estimate, reducing variance but potentially increasing bias. This allows practitioners to fine-tune the estimator based on the specific problem and the quality of the value function estimate.

Illustration of how bias decreases and variance increases as the GAE parameter $\lambda$ goes from 0 (TD advantage) to 1 (Monte Carlo advantage).

Practical Computation of GAE

The infinite sum in the GAE definition isn't practical for computation. Fortunately, we can compute it efficiently using a recursive formula, typically working backward from the end of a trajectory or a collected batch of experience of length $T$ .

Let $s_T$ be the last state in the sequence.

Calculate the TD errors for all steps $t = 0, 1, ..., T-1$ :
$\delta_t = r_{t+1} + \gamma V(s_{t+1}) - V(s_t)$
If $s_{t+1}$ is a terminal state, $V(s_{t+1})$ is defined as 0.
Compute the GAE estimates backward from $t = T-1$ down to $t = 0$ : Set $\hat{A}_T^{GAE} = 0$ if $s_T$ is terminal. If using bootstrapping after $T$ steps (i.e., the episode didn't terminate), you might calculate $\delta_T = r_{T+1} + \gamma V(s_{T+1}) - V(s_T)$ if data for $T+1$ is available, or simply set $\hat{A}_T^{GAE}=0$ and rely on the TD errors up to $T-1$ . A common approach for non-terminal states at the end of a batch is to use $\delta_{T-1}$ as $\hat{A}_{T-1}^{GAE}$ initially, or simply compute $\hat{A}_{T-1}^{GAE} = \delta_{T-1}$ . A slightly more precise way starts the recursion assuming $\hat{A}_T^{GAE} = 0$ .

Then, iterate backward:
$\hat{A}_t^{GAE} = \delta_t + \gamma \lambda \hat{A}_{t+1}^{GAE}$
Note that $V(s_t)$ and $V(s_{t+1})$ are provided by the current estimate from the critic network.

Why Use GAE?

GAE has become a standard component in many modern actor-critic algorithms, particularly TRPO and PPO. Its main advantage is the significant reduction in variance compared to Monte Carlo estimates, often leading to much more stable and faster learning, while controlling the bias introduced compared to simple TD error estimates.

The parameter $\lambda$ acts as a control knob. While $\lambda=1$ corresponds to the unbiased Monte Carlo estimate and $\lambda=0$ to the potentially biased TD(0) estimate, values in between (e.g., $\lambda=0.95$ or $\lambda=0.97$ ) often yield the best empirical performance by finding a good balance. Selecting the optimal $\lambda$ is problem-dependent and becomes another hyperparameter to tune during agent training.

In summary, Generalized Advantage Estimation provides a principled and effective method for estimating the advantage function in actor-critic algorithms. By introducing the $\lambda$ parameter, GAE offers a flexible mechanism to manage the important bias-variance trade-off, contributing significantly to the stability and performance of advanced policy gradient methods.

Was this section helpful?

References

High-Dimensional Continuous Control Using Generalized Advantage Estimation, John Schulman, Philipp Moritz, Sergey Levine, Michael Jordan, Pieter Abbeel, 2015 arXiv preprint arXiv:1506.02438 DOI: 10.48550/arXiv.1506.02438 - Introduces the Generalized Advantage Estimation (GAE) method, providing a principled approach to managing the bias-variance trade-off in policy gradient methods.
Reinforcement Learning: An Introduction, Richard S. Sutton and Andrew G. Barto, 2018 (MIT Press) - A comprehensive textbook that covers the fundamental concepts of reinforcement learning, including policy gradients, actor-critic methods, TD error, Monte Carlo methods, and value function estimation, providing the theoretical background for GAE.
Proximal Policy Optimization Algorithms, John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, Oleg Klimov, 2017 arXiv preprint arXiv:1707.06347 DOI: 10.48550/arXiv.1707.06347 - Presents the Proximal Policy Optimization (PPO) algorithm, a widely used actor-critic method that effectively utilizes Generalized Advantage Estimation (GAE) for stable and efficient policy updates.