Advantage Actor-Critic (A2C)

Advantage Actor-Critic (A2C) is a specific and popular implementation within the general Actor-Critic framework, where an actor learns the policy and a critic learns a value function. Standard policy gradient methods, such as REINFORCE, update policy parameters based on the total return $G_t$. While unbiased, using the full return often introduces high variance into the gradient estimates, leading to unstable and slow learning. The core idea behind A2C is to use a more informative signal than the raw return to guide policy updates.

Instead of simply asking "was this overall trajectory good or bad?", A2C focuses on "was this specific action $a_t$ taken in state $s_t$ better or worse than expected?". This relative measure is captured by the Advantage function, defined as:

$$A(s, a) = Q(s, a) - V(s)$$

Here, $Q(s, a)$ is the action-value function (the expected return after taking action $a$ in state $s$) and $V(s)$ is the state-value function (the expected return from state $s$ following the current policy). The advantage $A(s, a)$ quantifies how much better action $a$ is compared to the average action taken from state $s$ under the current policy $\pi$. If $A(s, a) > 0$, the action $a$ was better than average; if $A(s, a) < 0$, it was worse. Using the advantage as the scaling factor for policy updates provides a lower-variance signal compared to using just the return $G_t$ because the baseline $V(s)$ subtracts out the average value of the state, focusing the update on the relative quality of the chosen action.

Estimating the Advantage

In practice, we don't have access to the true $Q(s, a)$ and $V(s)$. A2C uses the critic's learned value function estimate, $V_\phi(s)$ (parameterized by $\phi$), to approximate the advantage. A common way to estimate the advantage for a transition $(s_t, a_t, r_{t+1}, s_{t+1})$ relies on the Temporal Difference (TD) error:

$$A(s_t, a_t) \approx r_{t+1} + \gamma V_\phi(s_{t+1}) - V_\phi(s_t)$$

Here, $r_{t+1} + \gamma V_\phi(s_{t+1})$ serves as an estimate of $Q(s_t, a_t)$ (the immediate reward plus the discounted estimated value of the next state). Subtracting the critic's estimate of the current state's value, $V_\phi(s_t)$, gives us the TD error, which acts as a one-step estimate of the advantage. While multi-step returns can sometimes provide better estimates, this single-step TD error is frequently used due to its simplicity and lower variance.

A2C Architecture and Updates

A common implementation pattern for A2C involves a neural network architecture where initial layers are shared between the actor and the critic, followed by separate output heads.

Diagram illustrating a typical A2C network with shared layers feeding into separate actor (policy) and critic (value) output heads.

The learning process involves updating both the actor and critic parameters, typically using gradient descent based on data collected from interacting with the environment.

Critic Update

The critic's parameters $\phi$ are updated to minimize the difference between its value estimate $V_\phi(s_t)$ and a target value, usually the TD target $r_{t+1} + \gamma V_\phi(s_{t+1})$. The loss function for the critic is typically the mean squared error:

$$L_{critic}(\phi) = \mathbb{E}_t \left[ (r_{t+1} + \gamma V_\phi(s_{t+1}) - V_\phi(s_t))^2 \right]$$

Minimizing this loss encourages the critic $V_\phi(s)$ to become a more accurate predictor of the expected discounted future rewards. Note that the target value $r_{t+1} + \gamma V_\phi(s_{t+1})$ uses the current critic parameters $\phi$; this is a form of bootstrapping.

Actor Update

The actor's parameters $\theta$ are updated using a policy gradient approach, but scaled by the advantage estimate derived from the critic. The objective is to increase the probability of actions that lead to positive advantages and decrease the probability of actions leading to negative advantages. The gradient ascent update direction for the actor is based on:

$$\nabla_\theta J(\theta) \approx \mathbb{E}_t \left[ \nabla_\theta \log \pi_\theta(a_t|s_t) \times A(s_t, a_t) \right]$$

where $A(s_t, a_t)$ is the estimated advantage, $r_{t+1} + \gamma V_\phi(s_{t+1}) - V_\phi(s_t)$. Notice that the advantage term $A(s_t, a_t)$ is treated as a constant when calculating the gradient with respect to $\theta$; we only differentiate the policy term $\log \pi_\theta(a_t|s_t)$. This update directly pushes the policy towards actions deemed better than average by the critic.

Combined Loss

In practice, the updates often happen simultaneously by minimizing a combined loss function:

$$L(\theta, \phi) = L_{actor}(\theta) + \beta \times L_{critic}(\phi)$$

where $L_{actor}(\theta) = - \mathbb{E}_t [ \log \pi_\theta(a_t|s_t) \times A(s_t, a_t) ]$ (note the negative sign for minimization), and $\beta$ is a hyperparameter weighting the critic loss (often set to 0.5 or 1.0). Sometimes, an entropy bonus term for the policy is added to the actor loss to encourage exploration, which can prevent the policy from converging prematurely to a suboptimal deterministic strategy.

Synchronous Updates

The standard A2C algorithm typically employs multiple parallel workers (actors) that collect experience from different copies of the environment simultaneously. These workers run for a fixed number of steps, calculate their respective advantage estimates and gradients, and then these gradients are aggregated (usually averaged) to perform a single, synchronous update on the shared model parameters ($\theta$ and $\phi$). This synchronous nature distinguishes it from its asynchronous counterpart, A3C, which we will discuss next. The synchronous updates can lead to more stable training compared to fully asynchronous methods, especially when using GPUs effectively.

By leveraging the critic's value estimates to compute the advantage, A2C provides a more stable and often more efficient learning signal for the actor compared to vanilla policy gradient methods like REINFORCE. It combines the benefits of policy-based methods (direct policy optimization, handling continuous actions) with the variance reduction techniques inspired by value-based methods.