ApX logoApX logo
ModelsCalculatorHardwareCoursesPricing
ApX logoApX logo
ModelsCalculatorHardwareCoursesPricing

All Courses

Introduction to Reinforcement Learning
Chapter 1: Foundations of Reinforcement Learning
What is Reinforcement Learning?
Agents and Environments
States, Actions, and Rewards
Policies: Mapping States to Actions
The RL Workflow: Interaction Loops
Types of RL Tasks: Episodic vs Continuing
Comparing RL with Other Learning Types
Setting up Your Python Environment for RL
Quiz for Chapter 1
Chapter 2: Markov Decision Processes (MDPs)
Modeling Sequential Decision Making
Formal Definition of an MDP
State Transition Probabilities
Reward Functions
Return: Cumulative Future Rewards
Discounting Future Rewards
Policies and Value Functions (Vπ, Qπ)
Finding Optimal Policies
Quiz for Chapter 2
Chapter 3: Estimating Value Functions
The Bellman Expectation Equation
The Bellman Optimality Equation
Solving Bellman Equations (Overview)
Dynamic Programming: Policy Iteration
Dynamic Programming: Value Iteration
Limitations of Dynamic Programming
Quiz for Chapter 3
Chapter 4: Monte Carlo Methods
Learning from Complete Episodes
Monte Carlo Prediction: Estimating Vπ
Monte Carlo Control: Estimating Qπ
On-Policy vs Off-Policy Learning
MC Control without Exploring Starts
On-Policy First-Visit MC Control Implementation
Off-Policy MC Prediction and Control Intro
Practice: Implementing MC Prediction
Quiz for Chapter 4
Chapter 5: Temporal-Difference Learning
Learning from Incomplete Episodes
TD(0) Prediction: Estimating Vπ
Advantages of TD Learning over MC
SARSA: On-Policy TD Control
Q-Learning: Off-Policy TD Control
Comparing SARSA and Q-Learning
Expected SARSA
Hands-on Practical: Implementing Q-Learning
Quiz for Chapter 5
Chapter 6: Function Approximation in RL
Handling Large State Spaces
Value Function Approximation (VFA)
Feature Vectors for State Representation
Linear Methods for VFA
Gradient Descent for Parameter Learning
Semi-gradient TD Methods
Using Neural Networks for VFA
Practice: Applying Linear VFA
Quiz for Chapter 6
Chapter 7: Introduction to Deep Q-Networks (DQN)
Combining Q-Learning with Deep Learning
Challenges with Neural Networks in RL
Experience Replay Mechanism
Fixed Q-Targets (Target Networks)
The DQN Algorithm Structure
Architectural Considerations for DQNs
Hands-on Practical: Building a Basic DQN
Quiz for Chapter 7
Chapter 8: Introduction to Policy Gradient Methods
Learning Policies Directly
Policy Gradient Theorem (Concept)
REINFORCE Algorithm
Baselines for Variance Reduction
Actor-Critic Methods Overview
Comparing Value-Based and Policy-Based Methods
Practice: Implementing REINFORCE
Quiz for Chapter 8

Baselines for Variance Reduction

The REINFORCE algorithm is a fundamental Monte Carlo policy gradient method. Understanding its update rule for the policy parameters θ\thetaθ is essential for effectively implementing baselines for variance reduction.

θ←θ+αGt∇θln⁡πθ(at∣st)\theta \leftarrow \theta + \alpha G_t \nabla_\theta \ln \pi_\theta(a_t|s_t)θ←θ+αGt​∇θ​lnπθ​(at​∣st​)

Here, GtG_tGt​ is the total discounted return starting from time step ttt. While this update rule correctly pushes the parameters in the direction of increased expected return, it often suffers from a significant drawback: high variance.

The Problem of High Variance

The return GtG_tGt​ is calculated based on a single, complete trajectory sampled according to the current policy πθ\pi_\thetaπθ​. The rewards received along this trajectory can vary substantially due to the stochastic nature of the environment and the policy itself. A slight change in an early action might lead to a completely different sequence of states and rewards, resulting in a very different GtG_tGt​.

This high variance in GtG_tGt​ translates directly into high variance in the gradient estimates ∇θln⁡πθ(at∣st)\nabla_\theta \ln \pi_\theta(a_t|s_t)∇θ​lnπθ​(at​∣st​). Imagine two different episodes starting from the same state sts_tst​. In one episode, the agent gets lucky and achieves a high return Gt=10G_t = 10Gt​=10. In another, perhaps due to an unlucky transition, the return is much lower, Gt=−5G_t = -5Gt​=−5. The resulting gradient updates for the policy parameters θ\thetaθ will point in very different directions, even if the action ata_tat​ taken was the same.

This noisy gradient signal makes learning inefficient and unstable. The parameter updates can oscillate wildly, requiring many samples (episodes) to average out the noise and make consistent progress. Consequently, the learning process can be extremely slow.

Introducing Baselines

To address this high variance, we can modify the policy gradient update by subtracting a baseline b(st)b(s_t)b(st​) from the return GtG_tGt​. The important requirement for the baseline is that it must not depend on the action ata_tat​; it should only depend on the state sts_tst​.

The modified update rule becomes:

θ←θ+α(Gt−b(st))∇θln⁡πθ(at∣st)\theta \leftarrow \theta + \alpha (G_t - b(s_t)) \nabla_\theta \ln \pi_\theta(a_t|s_t)θ←θ+α(Gt​−b(st​))∇θ​lnπθ​(at​∣st​)

Why does this help? Intuitively, we are no longer just increasing the probability of an action based on the raw return GtG_tGt​. Instead, we are comparing the obtained return GtG_tGt​ to an expected or average return b(st)b(s_t)b(st​) for being in state sts_tst​.

  • If Gt>b(st)G_t > b(s_t)Gt​>b(st​), the term (Gt−b(st))(G_t - b(s_t))(Gt​−b(st​)) is positive. This means the action ata_tat​ led to a better-than-expected outcome for state sts_tst​, so we increase its probability.
  • If Gt<b(st)G_t < b(s_t)Gt​<b(st​), the term (Gt−b(st))(G_t - b(s_t))(Gt​−b(st​)) is negative. The action ata_tat​ led to a worse-than-expected outcome, so we decrease its probability.

This comparison focuses the update on the relative quality of the action ata_tat​ given the state sts_tst​, rather than the absolute return GtG_tGt​, which can fluctuate greatly. By centering the returns around a baseline, we reduce the magnitude and variability of the update term, leading to lower variance in the gradient estimates and more stable learning.

Maintaining an Unbiased Gradient

A critical property of the baseline b(st)b(s_t)b(st​) is that subtracting it does not change the expected value of the gradient update. In other words, it reduces variance without introducing bias. We can show this mathematically. The expected value of the term we subtracted, scaled by the gradient log-probability, is zero:

Eπθ[b(st)∇θln⁡πθ(at∣st)]=∑sd(s)∑aπθ(a∣s)b(s)∇θln⁡πθ(a∣s)E_{\pi_\theta} [b(s_t) \nabla_\theta \ln \pi_\theta(a_t|s_t)] = \sum_s d(s) \sum_a \pi_\theta(a|s) b(s) \nabla_\theta \ln \pi_\theta(a|s)Eπθ​​[b(st​)∇θ​lnπθ​(at​∣st​)]=s∑​d(s)a∑​πθ​(a∣s)b(s)∇θ​lnπθ​(a∣s)

Using the identity ∇θln⁡πθ(a∣s)=∇θπθ(a∣s)πθ(a∣s)\nabla_\theta \ln \pi_\theta(a|s) = \frac{\nabla_\theta \pi_\theta(a|s)}{\pi_\theta(a|s)}∇θ​lnπθ​(a∣s)=πθ​(a∣s)∇θ​πθ​(a∣s)​, we get:

=∑sd(s)b(s)∑aπθ(a∣s)∇θπθ(a∣s)πθ(a∣s)= \sum_s d(s) b(s) \sum_a \pi_\theta(a|s) \frac{\nabla_\theta \pi_\theta(a|s)}{\pi_\theta(a|s)}=s∑​d(s)b(s)a∑​πθ​(a∣s)πθ​(a∣s)∇θ​πθ​(a∣s)​ =∑sd(s)b(s)∑a∇θπθ(a∣s)= \sum_s d(s) b(s) \sum_a \nabla_\theta \pi_\theta(a|s)=s∑​d(s)b(s)a∑​∇θ​πθ​(a∣s)

Since the sum of probabilities ∑aπθ(a∣s)\sum_a \pi_\theta(a|s)∑a​πθ​(a∣s) must equal 1 for any state sss, its gradient with respect to θ\thetaθ is zero:

=∑sd(s)b(s)∇θ∑aπθ(a∣s)= \sum_s d(s) b(s) \nabla_\theta \sum_a \pi_\theta(a|s)=s∑​d(s)b(s)∇θ​a∑​πθ​(a∣s) =∑sd(s)b(s)∇θ(1)=0= \sum_s d(s) b(s) \nabla_\theta (1) = 0=s∑​d(s)b(s)∇θ​(1)=0

Because the expected value of the subtracted term is zero, the expected value of the overall gradient remains unchanged. The baseline successfully reduces variance without altering the direction the policy parameters should move in, on average.

Choosing the Baseline: The State-Value Function

What makes a good baseline b(st)b(s_t)b(st​)? An effective and widely used choice is an estimate of the state-value function, V(st)V(s_t)V(st​). The state-value function V(st)V(s_t)V(st​) represents the expected return starting from state sts_tst​ and following the current policy πθ\pi_\thetaπθ​. This is precisely the kind of "expected outcome" we want to compare GtG_tGt​ against.

So, we use an approximation v^(st,w)\hat{v}(s_t, w)v^(st​,w), parameterized by weights www, as our baseline:

b(st)=v^(st,w)b(s_t) = \hat{v}(s_t, w)b(st​)=v^(st​,w)

The policy update then becomes:

θ←θ+α(Gt−v^(st,w))∇θln⁡πθ(at∣st)\theta \leftarrow \theta + \alpha (G_t - \hat{v}(s_t, w)) \nabla_\theta \ln \pi_\theta(a_t|s_t)θ←θ+α(Gt​−v^(st​,w))∇θ​lnπθ​(at​∣st​)

The term (Gt−v^(st,w))(G_t - \hat{v}(s_t, w))(Gt​−v^(st​,w)) is now an estimate of the advantage of taking action ata_tat​ in state sts_tst​. Recall that the advantage function is defined as A(s,a)=Q(s,a)−V(s)A(s, a) = Q(s, a) - V(s)A(s,a)=Q(s,a)−V(s). Since GtG_tGt​ is a Monte Carlo sample (an estimate) of Qπθ(st,at)Q^{\pi_\theta}(s_t, a_t)Qπθ​(st​,at​) (under policy πθ\pi_\thetaπθ​ for the specific action ata_tat​ taken) and v^(st,w)\hat{v}(s_t, w)v^(st​,w) estimates Vπθ(st)V^{\pi_\theta}(s_t)Vπθ​(st​), their difference approximates the advantage Aπθ(st,at)A^{\pi_\theta}(s_t, a_t)Aπθ​(st​,at​). Using the advantage estimate generally leads to lower variance than using the raw return GtG_tGt​.

How do we get the value function estimate v^(st,w)\hat{v}(s_t, w)v^(st​,w)? We need to learn it! This value function can be learned concurrently with the policy using methods we've already encountered, such as Monte Carlo prediction or Temporal-Difference learning (like TD(0)), often using techniques like function approximation (linear or neural networks) if the state space is large. This involves updating the value function parameters www based on the observed returns or TD errors.

This approach, where we learn both a policy (the "actor") and a value function (the "critic" which provides the baseline), forms the foundation for Actor-Critic methods, which we will introduce in the next section. By having the critic evaluate the actor's actions relative to the expected value of the state, we can significantly stabilize and accelerate policy gradient learning.

  • Reinforcement Learning: An Introduction, Richard S. Sutton and Andrew G. Barto, 2018 (MIT Press) - A standard textbook for reinforcement learning, offering a thorough explanation of policy gradient methods, variance reduction through baselines, and actor-critic algorithms.
  • Actor-Critic Algorithms, Vijay R. Konda and John N. Tsitsiklis, 2000 Advances in Neural Information Processing Systems, Vol. 12 (The MIT Press) - A paper that introduced and formalized the Actor-Critic framework, illustrating how a learned value function (critic) can serve as an effective baseline for policy gradient methods.
  • Spinning Up in Deep RL, Joshua Achiam, 2018-2023 (OpenAI) - An accessible online resource from OpenAI that provides practical explanations and implementations of policy gradient methods, including the application of baselines and advantage estimation in deep reinforcement learning.
  • High-Dimensional Continuous Control Using Generalized Advantage Estimation, John Schulman, Philipp Moritz, Sergey Levine, Michael Jordan, and Pieter Abbeel, 2016 International Conference on Learning Representations (ICLR 2016) (OpenReview) DOI: 10.48550/arXiv.1506.02438 - This paper introduces Generalized Advantage Estimation (GAE), a widely adopted technique for reducing variance in policy gradient methods by constructing improved advantage estimates, a direct application of baselines.

© 2026 ApX Machine Learning