ApX logoApX logo
ModelsCalculatorHardwareCoursesPricing
ApX logoApX logo
ModelsCalculatorHardwareCoursesPricing

All Courses

Advanced Reinforcement Learning Techniques
Chapter 1: Foundations Revisited and Function Approximation
Markov Decision Processes Formulation Review
Bellman Equations and Optimality Conditions
Value Iteration and Policy Iteration
Temporal Difference Learning Methods
Introduction to Policy Gradient Methods
Function Approximation in Reinforcement Learning
The Deadly Triad in Off-Policy Learning
Chapter 2: Deep Q-Networks and Enhancements
Limitations of Linear Function Approximation
Deep Q-Networks (DQN) Algorithm
Experience Replay Mechanism
Target Networks for Training Stability
Double Deep Q-Networks (DDQN)
Dueling Network Architectures
Prioritized Experience Replay (PER)
Distributional Reinforcement Learning Concepts
Rainbow DQN Integration
DQN Variants Implementation Hands-on Practical
Chapter 3: Advanced Policy Gradient and Actor-Critic Methods
Challenges in Basic Policy Gradients
Actor-Critic Architecture Fundamentals
Baselines for Variance Reduction
Advantage Actor-Critic (A2C) and A3C
Generalized Advantage Estimation (GAE)
Deep Deterministic Policy Gradient (DDPG)
Trust Region Policy Optimization (TRPO)
Proximal Policy Optimization (PPO)
Soft Actor-Critic (SAC)
Actor-Critic Methods Implementation Practice
Chapter 4: Advanced Exploration Strategies
The Exploration-Exploitation Trade-off Revisited
Optimism in the Face of Uncertainty: UCB Methods
Probability Matching: Thompson Sampling
Parameter Space Noise for Exploration
Pseudo-Counts: Count-Based Exploration
Prediction Error as Curiosity: Intrinsic Motivation
State Novelty: Random Network Distillation (RND)
Information Gain for Exploration
Comparing and Combining Exploration Techniques
Exploration Strategy Implementation Practice
Chapter 5: Model-Based Reinforcement Learning
Rationale for Model-Based RL
Taxonomy of Model-Based Methods
Learning Environment Dynamics Models
Dyna Architectures: Integrating Learning and Planning
Planning with Learned Models: Trajectory Sampling
Monte Carlo Tree Search (MCTS) Fundamentals
Integrating MCTS with Learned Models
Model Predictive Control (MPC) Connections
Challenges: Model Accuracy and Computational Cost
Simple Model-Based Agent Hands-on Practical
Chapter 6: Multi-Agent Reinforcement Learning
Introduction to Multi-Agent Systems
MARL Problem Formulation: Stochastic Games
Centralized vs Decentralized Control
Challenge: The Non-Stationarity Problem
Independent Learners (IQL, IDDPG)
Parameter Sharing Strategies
Centralized Training with Decentralized Execution (CTDE)
Value Decomposition Methods (VDN, QMIX)
Multi-Agent Deep Deterministic Policy Gradient (MADDPG)
Communication Protocols in MARL
MARL Implementation Practice
Chapter 7: Offline Reinforcement Learning
Introduction to Offline RL (Batch RL)
Differences from Online and Off-Policy RL
Challenge: Distributional Shift
Off-Policy Evaluation in the Offline Setting
Importance Sampling and its Limitations
Fitted Q-Iteration (FQI) Approaches
Policy Constraint Methods
Batch-Constrained Deep Q-learning (BCQ)
Value Regularization Methods
Conservative Q-Learning (CQL)
Offline RL Implementation Considerations
Offline RL Algorithm Practice
Chapter 8: Implementation Details and Optimization
Neural Network Architectures for RL
Hyperparameter Tuning Strategies
Action and Observation Space Representation
Code Structuring for RL Projects
Software Frameworks and Libraries
Distributed Reinforcement Learning Approaches
Reproducibility in Deep RL
Debugging and Visualization Techniques
Performance Optimization and Hardware Considerations
Agent Debugging and Tuning Practice

Baselines for Variance Reduction

The REINFORCE algorithm, while foundational for policy gradients, often struggles in practice due to the high variance inherent in its gradient estimates. The policy update relies on the total return GtG_tGt​ observed after taking action ata_tat​ in state sts_tst​. Because GtG_tGt​ can vary significantly depending on the subsequent stochastic transitions and actions, the resulting gradient estimates ∇θlog⁡πθ(at∣st)Gt\nabla_\theta \log \pi_\theta(a_t|s_t) G_t∇θ​logπθ​(at​∣st​)Gt​ can be noisy. This noise slows down learning and can prevent the policy from converging reliably to a good solution. Imagine trying to adjust a knob based on wildly fluctuating feedback; it's hard to know which direction is truly better.

Actor-Critic methods offer a compelling solution by introducing a baseline into the policy gradient calculation. The core idea is surprisingly simple: subtract a value from the return GtG_tGt​ that depends only on the state sts_tst​, let's call it b(st)b(s_t)b(st​). The modified policy gradient update term becomes:

∇θlog⁡πθ(at∣st)(Gt−b(st))\nabla_\theta \log \pi_\theta(a_t|s_t) (G_t - b(s_t))∇θ​logπθ​(at​∣st​)(Gt​−b(st​))

Why does this work? Crucially, subtracting a state-dependent baseline does not change the expected value of the policy gradient, meaning it doesn't introduce bias into the update direction. We can show this mathematically. The expected value of the term we subtract is:

Eπθ[∇θlog⁡πθ(at∣st)b(st)]=∑sdπθ(s)∑aπθ(a∣s)∇θlog⁡πθ(a∣s)b(s)\mathbb{E}_{\pi_\theta} \left[ \nabla_\theta \log \pi_\theta(a_t|s_t) b(s_t) \right] = \sum_s d^{\pi_\theta}(s) \sum_a \pi_\theta(a|s) \nabla_\theta \log \pi_\theta(a|s) b(s)Eπθ​​[∇θ​logπθ​(at​∣st​)b(st​)]=s∑​dπθ​(s)a∑​πθ​(a∣s)∇θ​logπθ​(a∣s)b(s)

Using the identity ∇θlog⁡πθ(a∣s)=∇θπθ(a∣s)πθ(a∣s)\nabla_\theta \log \pi_\theta(a|s) = \frac{\nabla_\theta \pi_\theta(a|s)}{\pi_\theta(a|s)}∇θ​logπθ​(a∣s)=πθ​(a∣s)∇θ​πθ​(a∣s)​, this becomes:

∑sdπθ(s)∑a∇θπθ(a∣s)b(s)=∑sdπθ(s)b(s)∇θ∑aπθ(a∣s)\sum_s d^{\pi_\theta}(s) \sum_a \nabla_\theta \pi_\theta(a|s) b(s) = \sum_s d^{\pi_\theta}(s) b(s) \nabla_\theta \sum_a \pi_\theta(a|s)s∑​dπθ​(s)a∑​∇θ​πθ​(a∣s)b(s)=s∑​dπθ​(s)b(s)∇θ​a∑​πθ​(a∣s)

Since ∑aπθ(a∣s)=1\sum_a \pi_\theta(a|s) = 1∑a​πθ​(a∣s)=1 for any state sss, its gradient with respect to θ\thetaθ is zero: ∇θ1=0\nabla_\theta 1 = 0∇θ​1=0. Therefore, the expected value of the subtracted term is zero:

Eπθ[∇θlog⁡πθ(at∣st)b(st)]=0\mathbb{E}_{\pi_\theta} \left[ \nabla_\theta \log \pi_\theta(a_t|s_t) b(s_t) \right] = 0Eπθ​​[∇θ​logπθ​(at​∣st​)b(st​)]=0

This confirms that subtracting a state-dependent baseline b(st)b(s_t)b(st​) maintains the unbiasedness of the policy gradient estimate.

How Baselines Reduce Variance

While the expected gradient remains the same, the variance of the gradient estimate can be significantly reduced. For instance, the term (Gt−b(st))(G_t - b(s_t))(Gt​−b(st​)): If we choose b(st)b(s_t)b(st​) to be a good estimate of the average return from state sts_tst​, then this term represents how much better or worse the actually observed return GtG_tGt​ was compared to the average expectation from that state.

  • If Gt>b(st)G_t > b(s_t)Gt​>b(st​): The action ata_tat​ led to a better-than-average outcome. The term is positive, reinforcing action ata_tat​.
  • If Gt<b(st)G_t < b(s_t)Gt​<b(st​): The action ata_tat​ led to a worse-than-average outcome. The term is negative, discouraging action ata_tat​.

By centering the returns around a state-specific average, the magnitude of the updates is scaled down. Instead of large positive updates based on high absolute returns (which might just come from a generally rewarding state), the updates focus on the relative quality of the action taken in that specific context. This leads to a more stable and often faster learning process.

Sample returns (GtG_tGt​) from a trajectory. Subtracting a baseline (here, the overall average return) centers the values used for policy updates around zero. Positive values correspond to returns better than the baseline, negative values to returns worse than the baseline. This centering helps reduce the update variance.

The State-Value Function as the Optimal Baseline

What is the best choice for b(st)b(s_t)b(st​)? While a simple constant baseline (like the average return over an episode) can help somewhat, a much more effective baseline is the state-value function, Vπθ(st)V^{\pi_\theta}(s_t)Vπθ​(st​). This function, by definition, represents the expected return starting from state sts_tst​ and following the current policy πθ\pi_\thetaπθ​.

Using V(st)V(s_t)V(st​) as the baseline, the update term becomes:

∇θlog⁡πθ(at∣st)(Gt−V(st))\nabla_\theta \log \pi_\theta(a_t|s_t) (G_t - V(s_t))∇θ​logπθ​(at​∣st​)(Gt​−V(st​))

The term (Gt−V(st))(G_t - V(s_t))(Gt​−V(st​)) is an estimate of the Advantage Function, Aπθ(st,at)A^{\pi_\theta}(s_t, a_t)Aπθ​(st​,at​). The advantage function measures how much better taking action ata_tat​ in state sts_tst​ is compared to the average action chosen by the policy πθ\pi_\thetaπθ​ from state sts_tst​. Formally:

Aπθ(st,at)=Qπθ(st,at)−Vπθ(st)A^{\pi_\theta}(s_t, a_t) = Q^{\pi_\theta}(s_t, a_t) - V^{\pi_\theta}(s_t)Aπθ​(st​,at​)=Qπθ​(st​,at​)−Vπθ​(st​)

where Qπθ(st,at)Q^{\pi_\theta}(s_t, a_t)Qπθ​(st​,at​) is the action-value function. Since GtG_tGt​ is a Monte Carlo sample estimate of Qπθ(st,at)Q^{\pi_\theta}(s_t, a_t)Qπθ​(st​,at​), the term (Gt−V(st))(G_t - V(s_t))(Gt​−V(st​)) serves as a sample estimate of the advantage Aπθ(st,at)A^{\pi_\theta}(s_t, a_t)Aπθ​(st​,at​).

Using the state-value function V(st)V(s_t)V(st​) as the baseline is theoretically shown to be the optimal choice (in the sense of minimizing the variance of the gradient estimate) among all functions that depend only on the state sts_tst​.

The Emergence of the Critic

This naturally leads us to the Actor-Critic architecture. We need a way to estimate V(st)V(s_t)V(st​) to use it as a baseline. This is precisely the role of the critic.

  • The Actor is responsible for learning and updating the policy parameters θ\thetaθ, typically using gradient ascent with the advantage estimate: θ←θ+α∇θlog⁡πθ(at∣st)A(st,at)\theta \leftarrow \theta + \alpha \nabla_\theta \log \pi_\theta(a_t|s_t) A(s_t, a_t)θ←θ+α∇θ​logπθ​(at​∣st​)A(st​,at​) (where A(st,at)A(s_t, a_t)A(st​,at​) is estimated, e.g., by Gt−V(st)G_t - V(s_t)Gt​−V(st​) or other methods we'll see later).
  • The Critic is responsible for learning an estimate of the state-value function V(st;ϕ)V(s_t; \phi)V(st​;ϕ) (or sometimes the action-value function Q(st,at;ϕ)Q(s_t, a_t; \phi)Q(st​,at​;ϕ)), parameterized by ϕ\phiϕ. It learns from the transitions experienced by the actor, often using Temporal Difference (TD) learning methods like TD(0). The critic's output V(st;ϕ)V(s_t; \phi)V(st​;ϕ) is then used by the actor as the baseline b(st)b(s_t)b(st​).

Diagram of an Actor-Critic architecture using V(s)V(s)V(s) as a baseline. The Actor selects actions based on the policy. The Critic evaluates states by learning V(s)V(s)V(s). The Critic's value estimate V(s)V(s)V(s) is used as a baseline to compute an Advantage estimate, which in turn provides a lower-variance signal for updating the Actor's policy. Both components learn from interactions with the environment.

In summary, introducing a state-dependent baseline, particularly the state-value function V(st)V(s_t)V(st​), is a powerful technique for reducing the variance of policy gradient estimates without introducing bias. This naturally motivates the Actor-Critic framework, where the critic learns the value function to provide this baseline, and the actor updates the policy using the resulting advantage signal. The following sections will explore specific algorithms like A2C/A3C that implement this idea effectively and introduce further refinements like Generalized Advantage Estimation (GAE).

  • Reinforcement Learning: An Introduction, Richard S. Sutton and Andrew G. Barto, 2018 (MIT Press) - A comprehensive textbook providing a detailed treatment of policy gradient methods, the role of baselines in variance reduction, and the principles of actor-critic architectures.
  • Policy Gradient Methods for Reinforcement Learning with Function Approximation, Richard S. Sutton, David A. McAllester, Satinder P. Singh, and Yishay Mansour, 1999 Advances in Neural Information Processing Systems, Vol. 12 (NeurIPS) DOI: 10.5555/3008751.3008851 - Introduces the policy gradient theorem and formally discusses how subtracting a state-dependent baseline, such as the value function, does not introduce bias while reducing variance in policy gradient estimates.
  • Asynchronous Methods for Deep Reinforcement Learning, Volodymyr Mnih, Adria Puigdomenech Badia, Mehdi Mirza, Alex Graves, Timothy Lillicrap, Tim Harley, David Silver, Koray Kavukcuoglu, 2016 Proceedings of The 33rd International Conference on Machine Learning, Vol. 48 (PMLR) - A seminal paper demonstrating successful deep reinforcement learning with an actor-critic architecture, utilizing the value function as a baseline to stabilize training and reduce variance through asynchronous updates.
  • High-Dimensional Continuous Control Using Generalized Advantage Estimation, John Schulman, Philipp Moritz, Sergey Levine, Michael Jordan, Pieter Abbeel, 2015 International Conference on Learning Representations (ICLR) DOI: 10.48550/arXiv.1506.02438 - Introduces Generalized Advantage Estimation (GAE), a method to achieve a good bias-variance trade-off in the advantage function estimate, which is important for advanced policy gradient methods.

© 2026 ApX Machine Learning