Combining Q-Learning with Deep Learning

Function approximation allows Reinforcement Learning agents to generalize knowledge across states, moving beyond the limitations of tabular methods for large state spaces. Although linear function approximation provides benefits, many challenges present highly complex relationships between states and values that linear models often struggle to capture. Consider tasks such as playing video games directly from screen pixels or controlling a robot based on high-dimensional sensor inputs. The state spaces in these scenarios are enormous, and the optimal policy might depend on intricate, non-linear patterns in the input.

This is where deep neural networks come into play. Neural networks excel at learning complex, hierarchical features and non-linear mappings from high-dimensional inputs. By using a neural network to approximate the action-value function $Q(s, a)$ , we can potentially learn effective policies even in environments with visually rich or highly complex state representations.

We call this approach Deep Q-Learning, and the neural network used is often referred to as a Deep Q-Network (DQN). Instead of storing Q-values in a table or using a simple linear function, we use a neural network with parameters (weights and biases) denoted collectively by $\theta$ . This network takes a state representation $s$ as input and outputs an estimate of the Q-value for each possible action $a$ in that state. A common architecture outputs a vector of Q-values, one entry for each action:

$Q(s, \cdot; \theta) \approx Q^*(s, \cdot)$

Here, $Q(s, \cdot; \theta)$ represents the network's output for state $s$ given parameters $\theta$ , which aims to approximate the true optimal action-value function $Q^*(s, \cdot)$ . For example, if an agent is processing an image from a game screen (the state $s$ ), the DQN might output the expected cumulative future reward for pressing left, pressing right, jumping, or firing.

The objective, then, is to train the network, meaning we need to find the parameters $\theta$ that make the network's output $Q(s, a; \theta)$ a good approximation of the optimal $Q^*(s, a)$ . How do we do this? We can leverage the principles of Q-learning and the Bellman optimality equation.

Recall the standard Q-learning update relies on the TD target: $y = R + \gamma \max_{a'} Q(S', a')$ . This target represents an estimate of the optimal Q-value based on the reward received ( $R$ ) and the maximum estimated future value from the next state ( $S'$ ). In Deep Q-Learning, we treat training the network as a supervised learning problem. For a given transition $(S, A, R, S')$ , the network predicts the current Q-value $Q(S, A; \theta)$ . We compute the target value $y$ using the reward $R$ and the network's own estimate of the maximum Q-value for the next state $S'$ .

Ideally, we want our network's prediction $Q(S, A; \theta)$ to match this target $y$ . We can define a loss function that measures the discrepancy between the prediction and the target. A common choice is the Mean Squared Error (MSE):

L(\theta) = \left( y - Q(S, A; \theta) \right)^2

L(\theta) = \left( \underbrace{R + \gamma \max_{a'} Q(S', a'; \theta)}_{\text{Target Q-value (y)}} - \underbrace{Q(S, A; \theta)}_{\text{Predicted Q-value}} \right)^2

Our goal is to minimize this loss function with respect to the network parameters $\theta$ . We can achieve this using gradient descent algorithms, such as Stochastic Gradient Descent (SGD) or its variants like Adam. By repeatedly sampling transitions $(S, A, R, S')$ and adjusting the network weights $\theta$ to reduce the loss, the network gradually learns to approximate the optimal action-value function.

This combination holds immense promise. It allows us to apply Q-learning to problems that were previously intractable due to the sheer size and complexity of their state spaces. However, applying neural networks directly within the standard RL loop introduces its own set of challenges. The sequential, correlated nature of data generated by an agent interacting with an environment violates the independence assumptions often made in supervised learning. Furthermore, the fact that the target value $y$ itself depends on the network parameters $\theta$ being updated can lead to instability during training. The following sections will discuss these challenges and introduce the core techniques developed for DQN, experience replay and target networks, which are designed to stabilize the learning process.

Was this section helpful?

References

Human-level control through deep reinforcement learning, Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Alex Graves, Ioannis Antonoglou, Daan Wierstra, and Martin Riedmiller, 2015 Nature, Vol. 518 DOI: 10.1038/nature14236 - Introduces the Deep Q-Network (DQN), combining Q-learning with deep neural networks for successful control tasks, establishing a key technique in deep reinforcement learning.
Reinforcement Learning: An Introduction, Richard S. Sutton and Andrew G. Barto, 2018 (MIT Press) - A standard textbook that explains reinforcement learning principles, including Q-learning, Bellman equations, and function approximation methods.
Deep Learning, Ian Goodfellow, Yoshua Bengio, and Aaron Courville, 2016 (MIT Press) - A foundational book providing background on deep neural networks, network architectures, and training using gradient-based optimization.