Intermediate Reinforcement Learning Techniques
Chapter 1: Revisiting Reinforcement Learning Fundamentals
The Reinforcement Learning Problem Setup
Markov Decision Processes (MDPs) Recap
Value Functions and Bellman Equations
Tabular Solution Methods: Q-Learning and SARSA
Limitations of Tabular Methods
Quiz for Chapter 1
Chapter 2: Deep Q-Networks (DQN)
Introduction to Function Approximation
Using Neural Networks for Q-Value Approximation
The DQN Algorithm Architecture
Experience Replay Mechanism
Fixed Q-Targets (Target Networks)
Loss Function for DQN Training
Hands-on Practical: Implementing DQN for CartPole
Quiz for Chapter 2
Chapter 3: Improvements and Variants of DQN
The Overestimation Problem in Q-Learning
Double DQN (DDQN)
Dueling Network Architectures
Combining DQN Improvements
Prioritized Experience Replay (Brief Overview)
Practice: Implementing Double DQN
Quiz for Chapter 3
Chapter 4: Policy Gradient Methods
Limitations of Value-Based Methods
Direct Policy Parameterization
The Policy Gradient Theorem
REINFORCE Algorithm
Understanding Variance in Policy Gradients
Baselines for Variance Reduction
Hands-on Practical: Implementing REINFORCE
Quiz for Chapter 4
Chapter 5: Actor-Critic Methods
Combining Policy and Value Estimation
Actor-Critic Architecture Overview
Advantage Actor-Critic (A2C)
Asynchronous Advantage Actor-Critic (A3C)
Implementation Considerations for Actor-Critic
Comparison: REINFORCE vs A2C/A3C
Practice: Developing an A2C Implementation
Quiz for Chapter 5
Double DQN (DDQN)
Was this section helpful?
- Deep Reinforcement Learning with Double Q-learning, Hado van Hasselt, Arthur Guez, David Silver, 2016 Proceedings of the AAAI Conference on Artificial Intelligence, Vol. 30 (Association for the Advancement of Artificial Intelligence) DOI: 10.1609/aaai.v30i1.10295 - This paper introduced the Double DQN algorithm to address overestimation bias in deep Q-networks.
- Reinforcement Learning: An Introduction, Richard S. Sutton and Andrew G. Barto, 2018 (The MIT Press) - A comprehensive textbook on reinforcement learning, including Q-learning, DQN, and its improvements.
- Human-level control through deep reinforcement learning, Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Alex Graves, Ioannis Antonoglou, Daan Wierstra, and Martin Riedmiller, 2015 Nature, Vol. 518 (Springer Nature) DOI: 10.1038/nature14236 - The original paper that introduced the Deep Q-Network (DQN), providing the foundational context for Double DQN.