ApX logoApX logo
ModelsCalculatorHardwareCoursesPricing
ApX logoApX logo
ModelsCalculatorHardwareCoursesPricing

All Courses

Intermediate Reinforcement Learning Techniques
Chapter 1: Revisiting Reinforcement Learning Fundamentals
The Reinforcement Learning Problem Setup
Markov Decision Processes (MDPs) Recap
Value Functions and Bellman Equations
Tabular Solution Methods: Q-Learning and SARSA
Limitations of Tabular Methods
Quiz for Chapter 1
Chapter 2: Deep Q-Networks (DQN)
Introduction to Function Approximation
Using Neural Networks for Q-Value Approximation
The DQN Algorithm Architecture
Experience Replay Mechanism
Fixed Q-Targets (Target Networks)
Loss Function for DQN Training
Hands-on Practical: Implementing DQN for CartPole
Quiz for Chapter 2
Chapter 3: Improvements and Variants of DQN
The Overestimation Problem in Q-Learning
Double DQN (DDQN)
Dueling Network Architectures
Combining DQN Improvements
Prioritized Experience Replay (Brief Overview)
Practice: Implementing Double DQN
Quiz for Chapter 3
Chapter 4: Policy Gradient Methods
Limitations of Value-Based Methods
Direct Policy Parameterization
The Policy Gradient Theorem
REINFORCE Algorithm
Understanding Variance in Policy Gradients
Baselines for Variance Reduction
Hands-on Practical: Implementing REINFORCE
Quiz for Chapter 4
Chapter 5: Actor-Critic Methods
Combining Policy and Value Estimation
Actor-Critic Architecture Overview
Advantage Actor-Critic (A2C)
Asynchronous Advantage Actor-Critic (A3C)
Implementation Considerations for Actor-Critic
Comparison: REINFORCE vs A2C/A3C
Practice: Developing an A2C Implementation
Quiz for Chapter 5

Advantage Actor-Critic (A2C)

Advantage Actor-Critic (A2C) is a specific and popular implementation within the general Actor-Critic framework, where an actor learns the policy and a critic learns a value function. Standard policy gradient methods, such as REINFORCE, update policy parameters based on the total return GtG_tGt​. While unbiased, using the full return often introduces high variance into the gradient estimates, leading to unstable and slow learning. The core idea behind A2C is to use a more informative signal than the raw return to guide policy updates.

Instead of simply asking "was this overall trajectory good or bad?", A2C focuses on "was this specific action ata_tat​ taken in state sts_tst​ better or worse than expected?". This relative measure is captured by the Advantage function, defined as:

A(s,a)=Q(s,a)−V(s)A(s, a) = Q(s, a) - V(s)A(s,a)=Q(s,a)−V(s)

Here, Q(s,a)Q(s, a)Q(s,a) is the action-value function (the expected return after taking action aaa in state sss) and V(s)V(s)V(s) is the state-value function (the expected return from state sss following the current policy). The advantage A(s,a)A(s, a)A(s,a) quantifies how much better action aaa is compared to the average action taken from state sss under the current policy π\piπ. If A(s,a)>0A(s, a) > 0A(s,a)>0, the action aaa was better than average; if A(s,a)<0A(s, a) < 0A(s,a)<0, it was worse. Using the advantage as the scaling factor for policy updates provides a lower-variance signal compared to using just the return GtG_tGt​ because the baseline V(s)V(s)V(s) subtracts out the average value of the state, focusing the update on the relative quality of the chosen action.

Estimating the Advantage

In practice, we don't have access to the true Q(s,a)Q(s, a)Q(s,a) and V(s)V(s)V(s). A2C uses the critic's learned value function estimate, Vϕ(s)V_\phi(s)Vϕ​(s) (parameterized by ϕ\phiϕ), to approximate the advantage. A common way to estimate the advantage for a transition (st,at,rt+1,st+1)(s_t, a_t, r_{t+1}, s_{t+1})(st​,at​,rt+1​,st+1​) relies on the Temporal Difference (TD) error:

A(st,at)≈rt+1+γVϕ(st+1)−Vϕ(st)A(s_t, a_t) \approx r_{t+1} + \gamma V_\phi(s_{t+1}) - V_\phi(s_t)A(st​,at​)≈rt+1​+γVϕ​(st+1​)−Vϕ​(st​)

Here, rt+1+γVϕ(st+1)r_{t+1} + \gamma V_\phi(s_{t+1})rt+1​+γVϕ​(st+1​) serves as an estimate of Q(st,at)Q(s_t, a_t)Q(st​,at​) (the immediate reward plus the discounted estimated value of the next state). Subtracting the critic's estimate of the current state's value, Vϕ(st)V_\phi(s_t)Vϕ​(st​), gives us the TD error, which acts as a one-step estimate of the advantage. While multi-step returns can sometimes provide better estimates, this single-step TD error is frequently used due to its simplicity and lower variance.

A2C Architecture and Updates

A common implementation pattern for A2C involves a neural network architecture where initial layers are shared between the actor and the critic, followed by separate output heads.

Diagram illustrating a typical A2C network with shared layers feeding into separate actor (policy) and critic (value) output heads.

The learning process involves updating both the actor and critic parameters, typically using gradient descent based on data collected from interacting with the environment.

Critic Update

The critic's parameters ϕ\phiϕ are updated to minimize the difference between its value estimate Vϕ(st)V_\phi(s_t)Vϕ​(st​) and a target value, usually the TD target rt+1+γVϕ(st+1)r_{t+1} + \gamma V_\phi(s_{t+1})rt+1​+γVϕ​(st+1​). The loss function for the critic is typically the mean squared error:

Lcritic(ϕ)=Et[(rt+1+γVϕ(st+1)−Vϕ(st))2]L_{critic}(\phi) = \mathbb{E}_t \left[ (r_{t+1} + \gamma V_\phi(s_{t+1}) - V_\phi(s_t))^2 \right]Lcritic​(ϕ)=Et​[(rt+1​+γVϕ​(st+1​)−Vϕ​(st​))2]

Minimizing this loss encourages the critic Vϕ(s)V_\phi(s)Vϕ​(s) to become a more accurate predictor of the expected discounted future rewards. Note that the target value rt+1+γVϕ(st+1)r_{t+1} + \gamma V_\phi(s_{t+1})rt+1​+γVϕ​(st+1​) uses the current critic parameters ϕ\phiϕ; this is a form of bootstrapping.

Actor Update

The actor's parameters θ\thetaθ are updated using a policy gradient approach, but scaled by the advantage estimate derived from the critic. The objective is to increase the probability of actions that lead to positive advantages and decrease the probability of actions leading to negative advantages. The gradient ascent update direction for the actor is based on:

∇θJ(θ)≈Et[∇θlog⁡πθ(at∣st)×A(st,at)]\nabla_\theta J(\theta) \approx \mathbb{E}_t \left[ \nabla_\theta \log \pi_\theta(a_t|s_t) \times A(s_t, a_t) \right]∇θ​J(θ)≈Et​[∇θ​logπθ​(at​∣st​)×A(st​,at​)]

where A(st,at)A(s_t, a_t)A(st​,at​) is the estimated advantage, rt+1+γVϕ(st+1)−Vϕ(st)r_{t+1} + \gamma V_\phi(s_{t+1}) - V_\phi(s_t)rt+1​+γVϕ​(st+1​)−Vϕ​(st​). Notice that the advantage term A(st,at)A(s_t, a_t)A(st​,at​) is treated as a constant when calculating the gradient with respect to θ\thetaθ; we only differentiate the policy term log⁡πθ(at∣st)\log \pi_\theta(a_t|s_t)logπθ​(at​∣st​). This update directly pushes the policy towards actions deemed better than average by the critic.

Combined Loss

In practice, the updates often happen simultaneously by minimizing a combined loss function:

L(θ,ϕ)=Lactor(θ)+β×Lcritic(ϕ)L(\theta, \phi) = L_{actor}(\theta) + \beta \times L_{critic}(\phi)L(θ,ϕ)=Lactor​(θ)+β×Lcritic​(ϕ)

where Lactor(θ)=−Et[log⁡πθ(at∣st)×A(st,at)]L_{actor}(\theta) = - \mathbb{E}_t [ \log \pi_\theta(a_t|s_t) \times A(s_t, a_t) ]Lactor​(θ)=−Et​[logπθ​(at​∣st​)×A(st​,at​)] (note the negative sign for minimization), and β\betaβ is a hyperparameter weighting the critic loss (often set to 0.5 or 1.0). Sometimes, an entropy bonus term for the policy is added to the actor loss to encourage exploration, which can prevent the policy from converging prematurely to a suboptimal deterministic strategy.

Synchronous Updates

The standard A2C algorithm typically employs multiple parallel workers (actors) that collect experience from different copies of the environment simultaneously. These workers run for a fixed number of steps, calculate their respective advantage estimates and gradients, and then these gradients are aggregated (usually averaged) to perform a single, synchronous update on the shared model parameters (θ\thetaθ and ϕ\phiϕ). This synchronous nature distinguishes it from its asynchronous counterpart, A3C, which we will discuss next. The synchronous updates can lead to more stable training compared to fully asynchronous methods, especially when using GPUs effectively.

By leveraging the critic's value estimates to compute the advantage, A2C provides a more stable and often more efficient learning signal for the actor compared to vanilla policy gradient methods like REINFORCE. It combines the benefits of policy-based methods (direct policy optimization, handling continuous actions) with the variance reduction techniques inspired by value-based methods.

  • Asynchronous Methods for Deep Reinforcement Learning, Volodymyr Mnih, Adrià Puigdomènech Badia, Mehdi Mirza, Alex Graves, Timothy P. Lillicrap, Tim Harley, David Silver, Koray Kavukcuoglu, 2016 ICML 2016 DOI: 10.48550/arXiv.1602.01783 - Introduces Asynchronous Advantage Actor-Critic (A3C), the asynchronous precursor to A2C, detailing the advantage function, shared network architecture, and parallel learning strategies.
  • Reinforcement Learning: An Introduction, Richard S. Sutton and Andrew G. Barto, 2018 (MIT Press) - Provides foundational concepts of Actor-Critic methods, policy gradients, value functions, and the use of baselines for variance reduction. See Chapter 13.
  • Spinning Up in Deep RL: Actor-Critic, Joshua Achiam, 2018 (OpenAI) - A practical guide explaining the implementation details of Actor-Critic algorithms, including A2C, its architecture, loss functions, and training considerations.

© 2026 ApX Machine Learning