ApX logoApX logo
ModelsCalculatorHardwareCoursesPricing
ApX logoApX logo
ModelsCalculatorHardwareCoursesPricing

All Courses

Intermediate Reinforcement Learning Techniques
Chapter 1: Revisiting Reinforcement Learning Fundamentals
The Reinforcement Learning Problem Setup
Markov Decision Processes (MDPs) Recap
Value Functions and Bellman Equations
Tabular Solution Methods: Q-Learning and SARSA
Limitations of Tabular Methods
Quiz for Chapter 1
Chapter 2: Deep Q-Networks (DQN)
Introduction to Function Approximation
Using Neural Networks for Q-Value Approximation
The DQN Algorithm Architecture
Experience Replay Mechanism
Fixed Q-Targets (Target Networks)
Loss Function for DQN Training
Hands-on Practical: Implementing DQN for CartPole
Quiz for Chapter 2
Chapter 3: Improvements and Variants of DQN
The Overestimation Problem in Q-Learning
Double DQN (DDQN)
Dueling Network Architectures
Combining DQN Improvements
Prioritized Experience Replay (Brief Overview)
Practice: Implementing Double DQN
Quiz for Chapter 3
Chapter 4: Policy Gradient Methods
Limitations of Value-Based Methods
Direct Policy Parameterization
The Policy Gradient Theorem
REINFORCE Algorithm
Understanding Variance in Policy Gradients
Baselines for Variance Reduction
Hands-on Practical: Implementing REINFORCE
Quiz for Chapter 4
Chapter 5: Actor-Critic Methods
Combining Policy and Value Estimation
Actor-Critic Architecture Overview
Advantage Actor-Critic (A2C)
Asynchronous Advantage Actor-Critic (A3C)
Implementation Considerations for Actor-Critic
Comparison: REINFORCE vs A2C/A3C
Practice: Developing an A2C Implementation
Quiz for Chapter 5

REINFORCE Algorithm

The Policy Gradient Theorem provides a way to calculate the gradient of the expected return with respect to the policy parameters θ\thetaθ. To apply this theory, a practical algorithm is required to estimate and use this gradient. The REINFORCE algorithm, also known as Monte Carlo Policy Gradient, offers precisely such an approach. It learns directly from complete episodes of experience, sampled using the current policy πθ\pi_\thetaπθ​.

The core idea is elegant: we want to adjust the policy parameters θ\thetaθ such that actions leading to higher overall returns become more probable, and actions leading to lower returns become less probable. REINFORCE achieves this by sampling trajectories and using the actual returns experienced within those trajectories to guide the parameter updates.

Recall the policy gradient theorem:

∇θJ(θ)=Eτ∼πθ[∑t=0T−1∇θlog⁡πθ(at∣st)Gt]\nabla_\theta J(\theta) = \mathbb{E}_{\tau \sim \pi_\theta} \left[ \sum_{t=0}^{T-1} \nabla_\theta \log \pi_\theta(a_t | s_t) G_t \right]∇θ​J(θ)=Eτ∼πθ​​[t=0∑T−1​∇θ​logπθ​(at​∣st​)Gt​]

Here, J(θ)J(\theta)J(θ) is the expected total return, τ\tauτ represents a trajectory (episode) sampled according to the policy πθ\pi_\thetaπθ​, GtG_tGt​ is the total discounted return starting from time step ttt, and ∇θlog⁡πθ(at∣st)\nabla_\theta \log \pi_\theta(a_t | s_t)∇θ​logπθ​(at​∣st​) is the gradient of the log-probability of taking action ata_tat​ in state sts_tst​. This term, often called the "score function", indicates the direction in parameter space that increases the probability of the action ata_tat​ taken at state sts_tst​.

Since we usually cannot compute the expectation Eτ∼πθ[⋅]\mathbb{E}_{\tau \sim \pi_\theta}[\cdot]Eτ∼πθ​​[⋅] directly (as it requires integrating over all possible trajectories), REINFORCE uses Monte Carlo estimation. We run the current policy πθ\pi_\thetaπθ​ to generate one or more complete episodes. For each episode, we calculate the return GtG_tGt​ for every time step ttt. Then, we approximate the gradient using the sampled data.

The REINFORCE Algorithm

The algorithm proceeds as follows:

  1. Initialize: Start with randomly initialized policy parameters θ\thetaθ.
  2. Loop (for each episode): a. Generate Trajectory: Run the policy πθ\pi_\thetaπθ​ in the environment to collect a full episode: τ=(s0,a0,r1,s1,a1,r2,...,sT−1,aT−1,rT)\tau = (s_0, a_0, r_1, s_1, a_1, r_2, ..., s_{T-1}, a_{T-1}, r_T)τ=(s0​,a0​,r1​,s1​,a1​,r2​,...,sT−1​,aT−1​,rT​). Here, TTT is the terminal time step of the episode. b. Loop (for each time step ttt from 000 to T−1T-1T−1): i. Calculate Return: Compute the return from time step ttt onwards: Gt=∑k=t+1Tγk−t−1rkG_t = \sum_{k=t+1}^{T} \gamma^{k-t-1} r_kGt​=∑k=t+1T​γk−t−1rk​. Note that GtG_tGt​ is the actual, observed discounted return following state sts_tst​ and action ata_tat​ in this specific trajectory. ii. Calculate Update: Determine the update direction for this step: ∇θlog⁡πθ(at∣st)Gt\nabla_\theta \log \pi_\theta(a_t | s_t) G_t∇θ​logπθ​(at​∣st​)Gt​. c. Update Parameters: Update the policy parameters using the accumulated gradients from the episode, typically using stochastic gradient ascent: θ←θ+α∑t=0T−1∇θlog⁡πθ(at∣st)Gt \theta \leftarrow \theta + \alpha \sum_{t=0}^{T-1} \nabla_\theta \log \pi_\theta(a_t | s_t) G_t θ←θ+α∑t=0T−1​∇θ​logπθ​(at​∣st​)Gt​ where α\alphaα is the learning rate. The update can also be performed after each step or after a batch of episodes.

Understanding the Update

Let's break down the update step θ←θ+α∇θlog⁡πθ(at∣st)Gt\theta \leftarrow \theta + \alpha \nabla_\theta \log \pi_\theta(a_t | s_t) G_tθ←θ+α∇θ​logπθ​(at​∣st​)Gt​.

  • ∇θlog⁡πθ(at∣st)\nabla_\theta \log \pi_\theta(a_t | s_t)∇θ​logπθ​(at​∣st​) (Score Function): This vector points in the direction in parameter space (θ\thetaθ) that most increases the log-probability of the action ata_tat​ that was actually taken in state sts_tst​ during the sampled episode.
  • GtG_tGt​ (Return): This scalar value represents the quality of the trajectory following the state-action pair (st,at)(s_t, a_t)(st​,at​). It acts as a weighting factor for the score function.
    • If GtG_tGt​ is large and positive, the update pushes θ\thetaθ significantly in the direction that makes ata_tat​ more likely in state sts_tst​. The algorithm "reinforces" this action because it led to a good outcome.
    • If GtG_tGt​ is large and negative (or small and positive), the update still pushes θ\thetaθ in the direction indicated by the score function, but the magnitude is smaller, or it might even push away from that direction if GtG_tGt​ is negative. The algorithm weakly reinforces or even discourages this action because it led to a poor outcome.

Essentially, REINFORCE correlates the direction of steepest ascent for the action probability (∇θlog⁡πθ\nabla_\theta \log \pi_\theta∇θ​logπθ​) with the return (GtG_tGt​) that followed that action. Actions associated with higher returns get "boosted" more strongly.

The following diagram outlines the information flow within a single REINFORCE update step based on a sampled episode.

Flow diagram illustrating how information from a sampled trajectory is used to calculate the return and score function, which are then combined to update the policy parameters in REINFORCE.

Strengths and Weaknesses

REINFORCE offers several advantages:

  • Simplicity: The core concept and update rule are relatively straightforward.
  • Direct Optimization: It directly optimizes the policy for the objective function (expected return).
  • Compatibility: Works naturally with both discrete and continuous action spaces (by parameterizing appropriate probability distributions). It can also learn optimal stochastic policies.

However, it also has significant drawbacks:

  • High Variance: The Monte Carlo estimation of the return GtG_tGt​ based on single trajectories can be extremely noisy. Different episodes can lead to significantly different returns, even if the policy changes slightly, resulting in high variance in the gradient estimate ∇θJ(θ)\nabla_\theta J(\theta)∇θ​J(θ). This high variance makes learning unstable and slow.
  • Sample Inefficiency: It typically requires many episodes to converge because of the noisy gradient estimates.
  • Requires Complete Episodes: Updates are performed only after an episode finishes, which can be problematic in very long or non-terminating tasks.

The high variance is the most pressing issue with the basic REINFORCE algorithm. In the next section, we will discuss how introducing baselines can help mitigate this problem significantly, leading to faster and more stable learning.

Implementation Approaches

When implementing REINFORCE, especially with neural networks:

  • Policy Representation: The policy πθ(a∣s)\pi_\theta(a|s)πθ​(a∣s) is typically represented by a neural network that takes the state sss as input.
    • For discrete actions, the network usually outputs probabilities for each action (e.g., using a softmax output layer). log⁡πθ(at∣st)\log \pi_\theta(a_t | s_t)logπθ​(at​∣st​) is then the log of the output probability corresponding to the action ata_tat​ that was taken.
    • For continuous actions, the network might output the parameters of a probability distribution (e.g., the mean μ\muμ and standard deviation σ\sigmaσ of a Gaussian distribution). The action ata_tat​ is sampled from N(μ(st;θ),σ(st;θ))\mathcal{N}(\mu(s_t; \theta), \sigma(s_t; \theta))N(μ(st​;θ),σ(st​;θ)), and log⁡πθ(at∣st)\log \pi_\theta(a_t | s_t)logπθ​(at​∣st​) is the log-probability density function (PDF) of the chosen action under that distribution.
  • Gradient Calculation: Deep learning libraries (like TensorFlow or PyTorch) handle the calculation of ∇θlog⁡πθ(at∣st)\nabla_\theta \log \pi_\theta(a_t | s_t)∇θ​logπθ​(at​∣st​) via automatic differentiation. You typically define a loss function whose gradient matches the policy gradient update. A common approach is to use a surrogate loss like −log⁡πθ(at∣st)Gt- \log \pi_\theta(a_t | s_t) G_t−logπθ​(at​∣st​)Gt​. Minimizing this loss using gradient descent is equivalent to performing gradient ascent on the objective J(θ)J(\theta)J(θ) using the term ∇θlog⁡πθ(at∣st)Gt\nabla_\theta \log \pi_\theta(a_t | s_t) G_t∇θ​logπθ​(at​∣st​)Gt​. Remember that GtG_tGt​ is treated as a constant factor when computing the gradient for the update at step ttt.

© 2026 ApX Machine Learning