Target Networks for Training Stability

Directly applying Q-learning updates with non-linear function approximators, such as deep neural networks, can lead to unstable training. The primary issue arises because the target value in the Temporal Difference (TD) error calculation depends on the same network parameters that are being updated. The standard Q-learning update target for a transition $(s_t, a_t, r_t, s_{t+1})$ is:

$$y_t = r_t + \gamma \max_{a'} Q(s_{t+1}, a'; \theta)$$

The loss function, often Mean Squared Error (MSE), would then be calculated using this target $y_t$ and the current Q-value estimate $Q(s_t, a_t; \theta)$:

$$L(\theta) = \mathbb{E} \left[ (y_t - Q(s_t, a_t; \theta))^2 \right] = \mathbb{E} \left[ (r_t + \gamma \max_{a'} Q(s_{t+1}, a'; \theta) - Q(s_t, a_t; \theta))^2 \right]$$

Notice that the parameters $\theta$ appear in both the target value calculation (through $\max_{a'} Q(s_{t+1}, a'; \theta)$) and the value being updated ($Q(s_t, a_t; \theta)$). When we compute the gradient and update $\theta$, the target $y_t$ also shifts. This "moving target" problem can cause oscillations or even divergence during training, as the network tries to chase a constantly changing objective.

The Target Network Solution

To mitigate this instability, the DQN algorithm introduced the concept of a target network. This is a separate neural network whose architecture is identical to the main Q-network (often called the online network or policy network) but whose parameters are kept frozen for a period.

Let's denote the parameters of the online network as $\theta$ and the parameters of the target network as $\theta^-$. The target network is used specifically to calculate the TD target value. The modified TD target becomes:

$$y_t = r_t + \gamma \max_{a'} Q(s_{t+1}, a'; \theta^-)$$

Now, the loss function is calculated as:

$$L(\theta) = \mathbb{E} \left[ (y_t - Q(s_t, a_t; \theta))^2 \right] = \mathbb{E} \left[ (r_t + \gamma \max_{a'} Q(s_{t+1}, a'; \theta^-) - Q(s_t, a_t; \theta))^2 \right]$$

Crucially, the parameters $\theta^-$ in the target calculation are held fixed during the gradient computation and update step for the online network parameters $\theta$. The online network $Q(s, a; \theta)$ is updated based on the stable targets provided by the target network $Q(s, a; \theta^-)$.

Updating the Target Network

The target network's parameters $\theta^-$ cannot remain fixed indefinitely, otherwise, the target values would become outdated and prevent the agent from learning improved Q-values. The standard approach in DQN is to periodically copy the parameters from the online network to the target network.

This update happens every $C$ training steps (where $C$ is a hyperparameter):

$$\theta^- \leftarrow \theta \quad \text{every } C \text{ steps}$$

Between these updates, $\theta^-$ remains constant, providing a stable target for the online network's updates over $C$ steps.

Flow showing the online network ($Q(s, a; \theta)$) being updated based on gradients from the loss, while the target network ($Q(s', a'; \theta^-)$) provides stable targets ($y_t$) for the loss calculation. The target network's parameters ($\theta^-$) are periodically updated by copying the online network's parameters ($\theta$).

Impact and Approaches

Using a target network significantly improves the stability of training deep Q-networks. By decoupling the target calculation from the immediately updated parameters, it prevents the destructive feedback loops that can arise from the "moving target" problem.

The target network update frequency, $C$, is an important hyperparameter.

• Smaller $C$: The target network updates more frequently, incorporating new Q-value information faster. However, this reduces stability as the target moves more often.
• Larger $C$: The target network updates less frequently, providing more stability. However, learning might slow down because the target values can become significantly outdated ("stale") relative to the rapidly changing online network.

Typical values for $C$ in practice range from hundreds to tens of thousands of update steps, depending on the environment and specific algorithm configuration. Finding a suitable value often requires empirical tuning.

Soft Target Updates

An alternative to the periodic "hard" copy is the "soft" update, also known as Polyak averaging. This method updates the target network parameters slowly towards the online network parameters at every training step:

$$\theta^- \leftarrow \tau \theta + (1 - \tau) \theta^-$$

Here, $\tau$ is a small constant (e.g., $\tau = 0.001$ or $\tau = 0.005$). This approach results in a target network that changes more smoothly over time compared to the periodic hard updates. While originally popularized in algorithms like Deep Deterministic Policy Gradient (DDPG) for continuous control, soft updates can also be used with DQN variants.

In summary, target networks are a foundational technique for stabilizing deep Q-learning. By using a periodically updated copy of the main network to generate TD targets, they break the correlation that leads to instability, allowing deep neural networks to be trained effectively for complex reinforcement learning tasks.