Architectural Considerations for DQNs

Building the neural network that approximates the action-value function, $Q(s, a; \theta)$, is a primary consideration in Deep Q-Networks (DQN). DQNs utilize core mechanisms such as experience replay and target networks. The design of this network is highly dependent on the nature of the state representation provided by the environment.

Network Input: State Representation

The first step is determining the input layer's structure. What does the agent "see"?

• Vector-Based States: If the state is represented as a vector of numerical features (e.g., positions, velocities, sensor readings), a standard Multilayer Perceptron (MLP), also known as a fully connected network, is often sufficient. The input layer will have a number of neurons equal to the number of features in the state vector.

• Image-Based States: For environments where the state is represented by images (like pixels from a screen), Convolutional Neural Networks (CNNs) are the standard choice. CNNs are adept at recognizing spatial hierarchies and patterns in grid-like data. The input layer will typically accept the dimensions of the image (height, width, color channels). Often, multiple consecutive frames are stacked together as input to provide the network with temporal information, like velocity or direction of movement.

Network Architecture: Hidden Layers

Following the input layer, one or more hidden layers process the information.

• MLPs: Hidden layers in an MLP are typically fully connected layers. Each neuron in a layer receives input from all neurons in the previous layer. The number of layers and the number of neurons in each layer are hyperparameters you need to choose. Start simple (e.g., one or two hidden layers with a moderate number of neurons) and increase complexity if needed. The Rectified Linear Unit (ReLU) activation function ($f(x) = max(0, x)$) is a common and effective choice for hidden layers in DQNs.

• CNNs: For image inputs, the initial hidden layers are typically convolutional layers followed by pooling layers. Convolutional layers apply filters to detect local patterns (edges, textures), while pooling layers reduce the spatial dimensions, making the representation more manageable and invariant to small translations. After several convolutional and pooling layers, the resulting feature maps are usually flattened into a vector and fed into one or more fully connected layers, similar to an MLP. Again, ReLU is a standard activation function for these layers. The famous DQN paper that mastered Atari games used a CNN architecture consisting of several convolutional layers followed by fully connected layers.

Network Output: Action Values

The final layer of the network is important: it must output the estimated Q-values.

• Structure: The output layer is typically a fully connected layer with one neuron for each possible discrete action the agent can take in the environment.
• Activation: Since Q-values represent expected cumulative rewards and are not restricted to a specific range (like probabilities), the output layer typically uses a linear activation function (i.e., no activation function or an identity function).
• Interpretation: The value produced by the $i$-th output neuron represents the network's estimate of the Q-value for taking the $i$-th action in the input state $s$. That is, the output vector approximates $Q(s, a; \theta)$ for all possible actions $a$ simultaneously.

Example Architectures

Here are diagrams illustrating common structures:

1. MLP for Vector States:

An MLP takes a state vector $s = (s_1, ..., s_n)$ and passes it through fully connected (FC) hidden layers (often with ReLU activations) to produce Q-value estimates for each action $a_1, ..., a_m$.

2. CNN for Image States (Simplified):

A CNN processes input image(s) through convolutional and pooling layers to extract spatial features. These features are then flattened and passed through fully connected layers to output Q-values for each action.

General Recommendations

• Start Simple: Begin with a relatively shallow network (few layers) and fewer neurons/filters. You can increase complexity later if performance is lacking. Overly complex networks can be harder and slower to train.
• Match Complexity to Problem: Simple environments (like CartPole) might only require a small MLP, while complex environments with visual input (like Atari games) necessitate deeper CNNs.
• Use Standard Practices: Employ common activation functions like ReLU for hidden layers and linear for the output. Consider using established architectures from research papers (like the original DQN Atari architecture) as a starting point.
• Iterate: Network design often involves experimentation. Train your agent, evaluate its performance, and adjust the architecture (number of layers, neurons, filter sizes, etc.) based on the results.

Choosing the right architecture is a significant part of applying DQN successfully. By considering the nature of your state space and leveraging common network design patterns, you can build effective function approximators for your reinforcement learning agents.