Using Neural Networks for VFA

Linear function approximation provides a way to generalize value estimates across states using features. However, complex, non-linear relationships often exist between a situation (state) and its long-term value in many interesting simulated environments. A simple linear combination of features might not be expressive enough to capture these intricate patterns. For instance, the value of a state might depend on subtle interactions between features that linear models struggle to represent.

This is where Neural Networks (NNs) come into play. NNs are powerful function approximators, renowned for their ability to learn complex, non-linear mappings directly from data. Their success in fields like computer vision and natural language processing stems from this capability. In Reinforcement Learning, we can leverage NNs to approximate value functions, potentially leading to much better performance in complex environments.

Approximating Value Functions with Neural Networks

Instead of using a linear function $\hat{v}(s, \mathbf{w}) = \mathbf{w}^T \mathbf{x}(s)$, we can use a neural network. The network takes the state representation as input and outputs the estimated value. Let $\mathbf{w}$ now denote the entire set of weights and biases within the neural network.

• For State-Value Functions: The network takes the state $s$ (or its feature vector $\mathbf{x}(s)$) as input and outputs a single scalar value, representing the estimated state value $\hat{v}(s; \mathbf{w})$.
• For Action-Value Functions: The network typically takes the state $s$ as input and outputs multiple values, one for each possible action $a$. This output represents the estimated Q-values $\hat{q}(s, a; \mathbf{w})$ for all actions available in state $s$. Alternatively, the network could take both the state $s$ and a specific action $a$ as input and output a single Q-value for that pair. The former approach is more common in methods like DQN.

A diagram showing a neural network taking a state representation as input and outputting estimated Q-values for multiple actions. The weights $\mathbf{w}$ parameterize the connections within the network.

Advantages of Using Neural Networks

1. Capturing Non-Linearity: NNs with hidden layers and non-linear activation functions (like ReLU) can approximate arbitrarily complex functions. This allows them to model sophisticated value functions that are more complex than linear methods.
2. Automatic Feature Representation: Deep neural networks, particularly Convolutional Neural Networks (CNNs) when dealing with image inputs (like game screens), can learn hierarchical features directly from raw data. This reduces or eliminates the need for manual, domain-specific feature engineering, which can be a significant bottleneck in applying RL. The network learns the features that are most predictive of value.

Training Neural Networks for Value Function Approximation

The fundamental goal remains the same as with linear VFA: adjust the parameters $\mathbf{w}$ (now the network's weights) to minimize the difference between the predicted value and a target value. We typically use variants of Temporal Difference (TD) learning.

For example, in a Q-learning context using an NN, the target value for an experience tuple $(S_t, A_t, R_{t+1}, S_{t+1})$ is often:

$$Y_t = R_{t+1} + \gamma \max_{a'} \hat{q}(S_{t+1}, a', \mathbf{w})$$

The network predicts $\hat{q}(S_t, A_t, \mathbf{w})$. The objective is to minimize the squared error between the target and the prediction, often called the TD error: $\delta_t = Y_t - \hat{q}(S_t, A_t, \mathbf{w})$.

We update the weights $\mathbf{w}$ using stochastic gradient descent (SGD) or its variants (like Adam). The update aims to move the prediction closer to the target:

$$\mathbf{w} \leftarrow \mathbf{w} + \alpha \delta_t \nabla_{\mathbf{w}} \hat{q}(S_t, A_t, \mathbf{w})$$

Here, $\nabla_{\mathbf{w}} \hat{q}(S_t, A_t, \mathbf{w})$ is the gradient of the network's output (for the specific action $A_t$) with respect to its weights $\mathbf{w}$. This gradient is calculated efficiently using the backpropagation algorithm, a standard technique in deep learning. Thankfully, modern deep learning libraries like TensorFlow or PyTorch handle the automatic differentiation and backpropagation for us. We only need to define the network architecture and the loss function (typically Mean Squared Error based on the TD error).

Note that this is still a semi-gradient method because the target $Y_t$ itself depends on the current weights $\mathbf{w}$ (unless using a target network, discussed later), and we don't differentiate through the target calculation when computing the gradient.

Challenges and What's Next

While powerful, directly combining NNs with TD learning introduces potential instability during training. Two main issues arise:

1. Correlated Data: RL experiences $(S_t, A_t, R_{t+1}, S_{t+1})$ are sequential and highly correlated. This violates the assumption of independent and identically distributed (IID) samples that standard SGD relies upon, potentially leading to inefficient or unstable learning.
2. Moving Target: The TD target $Y_t$ depends on the network's own estimates $\hat{q}(S_{t+1}, a', \mathbf{w})$. Since the weights $\mathbf{w}$ are updated at each step, the target value itself keeps changing. It's like chasing a moving target, which can cause oscillations or divergence.

Furthermore, NNs introduce more hyperparameters (network architecture, layer sizes, learning rates, activation functions) that require careful selection and tuning.

Using neural networks for value function approximation marks the transition towards Deep Reinforcement Learning (DRL). The next chapter, "Introduction to Deep Q-Networks (DQN)", will directly address the stability challenges mentioned above by introducing techniques like Experience Replay and Fixed Q-Targets, which were instrumental in the success of early DRL algorithms. These techniques allow us to effectively train deep neural networks for complex RL tasks.