Intermediate Reinforcement Learning Techniques
Chapter 1: Revisiting Reinforcement Learning Fundamentals
The Reinforcement Learning Problem Setup
Markov Decision Processes (MDPs) Recap
Value Functions and Bellman Equations
Tabular Solution Methods: Q-Learning and SARSA
Limitations of Tabular Methods
Quiz for Chapter 1
Chapter 2: Deep Q-Networks (DQN)
Introduction to Function Approximation
Using Neural Networks for Q-Value Approximation
The DQN Algorithm Architecture
Experience Replay Mechanism
Fixed Q-Targets (Target Networks)
Loss Function for DQN Training
Hands-on Practical: Implementing DQN for CartPole
Quiz for Chapter 2
Chapter 3: Improvements and Variants of DQN
The Overestimation Problem in Q-Learning
Double DQN (DDQN)
Dueling Network Architectures
Combining DQN Improvements
Prioritized Experience Replay (Brief Overview)
Practice: Implementing Double DQN
Quiz for Chapter 3
Chapter 4: Policy Gradient Methods
Limitations of Value-Based Methods
Direct Policy Parameterization
The Policy Gradient Theorem
REINFORCE Algorithm
Understanding Variance in Policy Gradients
Baselines for Variance Reduction
Hands-on Practical: Implementing REINFORCE
Quiz for Chapter 4
Chapter 5: Actor-Critic Methods
Combining Policy and Value Estimation
Actor-Critic Architecture Overview
Advantage Actor-Critic (A2C)
Asynchronous Advantage Actor-Critic (A3C)
Implementation Considerations for Actor-Critic
Comparison: REINFORCE vs A2C/A3C
Practice: Developing an A2C Implementation
Quiz for Chapter 5
Implementation Considerations for Actor-Critic
Was this section helpful?
- Reinforcement Learning: An Introduction, Richard S. Sutton and Andrew G. Barto, 2018 (MIT Press) - Comprehensive textbook on reinforcement learning, detailing Actor-Critic methods, policy gradient theory, and value function approximation.
- Asynchronous Methods for Deep Reinforcement Learning, Volodymyr Mnih, Adrià Puigdomènech Badia, Mehdi Mirza, Alex Graves, Timothy P. Lillicrap, Tim Harley, David Silver, Koray Kavukcuoglu, 2016 International Conference on Machine Learning (ICML) DOI: 10.48550/arXiv.1602.01783 - Introduces the A3C algorithm, featuring asynchronous updates, shared network architectures, and entropy regularization for deep Actor-Critic agents.
- High-Dimensional Continuous Control Using Generalized Advantage Estimation, John Schulman, Philipp Moritz, Sergey Levine, Michael Jordan, Pieter Abbeel, 2016 International Conference on Learning Representations (ICLR) DOI: 10.48550/arXiv.1506.02438 - Presents Generalized Advantage Estimation (GAE), a method to estimate the advantage term ($A_t$) with reduced variance in policy gradient methods, commonly used in Actor-Critic.
- Proximal Policy Optimization Algorithms, John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, Oleg Klimov, 2017 arXiv preprint DOI: 10.48550/arXiv.1707.06347 - Describes PPO, a stable and widely adopted policy gradient algorithm that uses an Actor-Critic architecture with a clipped objective.