The DQN Algorithm Architecture

Tabular Q-learning struggles with environments that have an enormous number of states. Imagine trying to store a Q-value for every possible configuration of pixels on a screen; it is not feasible. Deep Q-Networks (DQN) solve this by replacing the Q-table with a deep neural network, often called the Q-network. This network acts as a function approximator, learning to estimate the Q-values.

The Q-Network Structure

Instead of looking up a value in a table, we feed the agent's current state representation, $s$, into the neural network. The network's job is to output an estimate of the action-value function, $Q(s, a)$, for all possible actions $a$ available in that state $s$.

• Input: The input layer of the network receives the state representation. This could be raw pixel data from a game screen, sensor readings from a robot, or other features describing the environment's current condition. Preprocessing might be needed to normalize or structure this input appropriately for the network.
• Hidden Layers: These are standard neural network layers (e.g., convolutional layers for image inputs, fully connected layers) that process the input state and learn complex features and relationships relevant to estimating action values.
• Output: The output layer produces a vector of values. Each element in this vector corresponds to the estimated Q-value for one specific action. For example, if an environment has 4 possible actions (up, down, left, right), the network will output 4 values: $Q(s, \text{up})$, $Q(s, \text{down})$, $Q(s, \text{left})$, and $Q(s, \text{right})$.

Let $\theta$ represent the parameters (weights and biases) of this neural network. We can denote the network's output for state $s$ and action $a$ as $Q(s, a; \theta)$.

Diagram illustrating the basic flow within the DQN agent at decision time. The current state $s_t$ is fed into the Q-network, which outputs estimated Q-values for all actions. An action selection strategy (like epsilon-greedy) then uses these Q-values to choose the action $a_t$ to take in the environment.

The Learning Process: Adapting Q-Learning

How do we train this Q-network? We adapt the core idea from the Q-learning update rule. Recall the Bellman equation, which expresses the relationship between the Q-value of the current state-action pair and the Q-values of the next state:

$$Q^*(s, a) = \mathbb{E}[R_{t+1} + \gamma \max_{a'} Q^*(S_{t+1}, a') | S_t=s, A_t=a]$$

In standard Q-learning, we iteratively update our Q-table entries towards this target value. With DQN, we frame this as a supervised learning problem. We want our network's prediction, $Q(S_t, A_t; \theta)$, to approximate a target value derived from the Bellman equation.

For a given transition $(S_t, A_t, R_{t+1}, S_{t+1})$, the target value, $y_t$, is calculated using the reward received and the maximum estimated Q-value for the next state, $S_{t+1}$:

$$y_t = R_{t+1} + \gamma \max_{a'} Q(S_{t+1}, a'; \theta)$$

Notice something important here: in this basic formulation, we are using the same Q-network (with parameters $\theta$) both to predict the Q-value for the current state-action pair ($Q(S_t, A_t; \theta)$) and to estimate the maximum Q-value for the next state ($\max_{a'} Q(S_{t+1}, a'; \theta)$) needed to compute the target $y_t$. As we will see later, this coupling can lead to instabilities during training.

The Loss Function

The network learns by minimizing the difference between its prediction $Q(S_t, A_t; \theta)$ and the target value $y_t$. A common choice for the loss function is the Mean Squared Error (MSE), averaged over a batch of transitions:

$$L(\theta) = \mathbb{E} \left[ (y_t - Q(S_t, A_t; \theta))^2 \right]$$

Substituting the definition of $y_t$:

$$L(\theta) = \mathbb{E} \left[ \left( R_{t+1} + \gamma \max_{a'} Q(S_{t+1}, a'; \theta) - Q(S_t, A_t; \theta) \right)^2 \right]$$

We then use standard gradient descent algorithms (like Adam or RMSprop) to compute the gradient of this loss function with respect to the network parameters $\theta$, $\nabla_{\theta} L(\theta)$, and update the weights to minimize the error:

$$\theta \leftarrow \theta - \alpha \nabla_{\theta} L(\theta)$$

where $\alpha$ is the learning rate.

This architecture allows the agent to learn Q-values even in high-dimensional state spaces by leveraging the representational power of deep neural networks. However, training this setup directly often encounters significant challenges related to correlated samples and moving target values. The next sections will introduce two fundamental techniques, Experience Replay and Target Networks, designed specifically to stabilize the training process of Deep Q-Networks.