Fixed Q-Targets (Target Networks)

Experience replay helps break the correlation between consecutive training samples, improving training stability. However, training Q-networks encounters another primary source of instability: the constantly changing target value used in the Temporal Difference (TD) update.

Recall the standard Q-learning update rule applied to function approximation. We aim to minimize the difference between our current Q-value estimate $Q(S_t, A_t; \theta)$ and a target value. This target is typically calculated using the reward $R_{t+1}$ and the estimated value of the next state, $S_{t+1}$. In Q-learning, this involves finding the maximum Q-value for the next state using the current network parameters $\theta$:

$$\text{Target}_t = R_{t+1} + \gamma \max_{a'} Q(S_{t+1}, a'; \theta_t)$$

The loss function, often the Mean Squared Error (MSE), would then be something like:

$$\mathcal{L}(\theta_t) = \mathbb{E} \left[ \left( (R_{t+1} + \gamma \max_{a'} Q(S_{t+1}, a'; \theta_t)) - Q(S_t, A_t; \theta_t) \right)^2 \right]$$

Notice that the parameters $\theta_t$ appear in both the target calculation and the value we are trying to adjust ($Q(S_t, A_t; \theta_t)$). When we perform gradient descent to update $\theta_t$, we are essentially chasing a moving target. As the network's weights $\theta_t$ change in each step, the target value itself shifts. This interdependence can lead to oscillations or even divergence during training, making learning unstable. It's like trying to hit a target that moves every time you adjust your aim.

Stabilizing Targets with a Target Network

To address this "moving target" problem, the DQN algorithm introduces a second neural network: the target network. This target network, let's denote its parameters by $\theta^-$, is essentially a clone of the online Q-network (the one we are actively training, with parameters $\theta$).

Here's how it works:

1. Initialization: The target network parameters $\theta^-$ are initialized to be identical to the online network parameters $\theta$.
2. Target Calculation: When calculating the TD target for the loss function, we use the target network $\theta^-$ to estimate the value of the next state. The target value $y_t$ becomes: $$y_t = R_{t+1} + \gamma \max_{a'} Q(S_{t+1}, a'; \theta^-)$$ Note the use of $\theta^-$ instead of $\theta_t$.
3. Loss Calculation: The loss is then calculated using this fixed target $y_t$ and the online network's prediction for the current state-action pair: $$\mathcal{L}(\theta_t) = \mathbb{E}_{(S_t, A_t, R_{t+1}, S_{t+1}) \sim D} \left[ \left( y_t - Q(S_t, A_t; \theta_t) \right)^2 \right]$$ Here, $D$ represents the experience replay buffer from which transitions are sampled.
4. Parameter Update: Only the online network parameters $\theta_t$ are updated via gradient descent using this loss. The target network parameters $\theta^-$ remain frozen during these updates.
5. Periodic Updates: After a fixed number of training steps (let's call this frequency $C$), the weights of the online network are copied over to the target network: $\theta^- \leftarrow \theta_t$.

This mechanism provides stability because the target value $y_t$ remains fixed for $C$ consecutive updates of the online network $\theta$. The online network is now learning to approximate a stationary target, which significantly simplifies the learning dynamics and reduces the likelihood of oscillations and divergence. The frequency $C$ of updating the target network is a hyperparameter that needs to be chosen; typical values might range from hundreds to thousands of steps, depending on the specific problem.

Interaction flow showing how the online network ($Q(S_t, A; \theta)$) and the target network ($Q(S_{t+1}, a'; \theta^-)$) are used during a DQN update step. The online network parameters $\theta$ are updated frequently via gradient descent, while the target network parameters $\theta^-$ are updated only periodically by copying from $\theta$.

By combining fixed Q-targets with experience replay, DQN addresses two major sources of instability inherent in applying Q-learning with complex function approximators like deep neural networks. Experience replay decorrelates the data samples, and target networks provide stable targets for the learning updates. Together, these techniques were fundamental to the success of early DQNs in learning to play Atari games from raw pixel inputs.