Implementation Considerations for Actor-Critic

Implementing Actor-Critic methods, characterized by their separate Actor and Critic components, involves several design choices that significantly impact performance and stability. The practical aspects encountered when constructing these agents are examined.

Shared vs. Separate Network Parameters

A fundamental architectural decision is whether the Actor and Critic should use entirely separate neural networks or share some layers.

• Separate Networks: The Actor ($\pi_\theta(a|s)$) and Critic ($V_\phi(s)$ or $Q_\phi(s, a)$) are implemented as distinct neural networks, each with its own set of parameters ($\theta$ and $\phi$, respectively). They are optimized independently using their respective loss functions.
• Shared Networks: A common approach, especially for high-dimensional inputs like images, is to have the networks share initial layers (e.g., convolutional layers for feature extraction) and then branch out into separate heads for the policy output (Actor) and value output (Critic).

Comparison of shared and separate network architectures for Actor-Critic methods. Shared networks can improve efficiency, while separate networks offer more independence.

Advantages of Shared Networks:

• Parameter Efficiency: Reduces the total number of parameters, potentially speeding up training and reducing memory usage.
• Feature Sharing: The shared layers learn representations useful for both predicting the value of a state and deciding on the best action in that state. This shared learning can sometimes lead to better performance.

Disadvantages of Shared Networks:

• Optimization Interference: The gradients from the Actor loss and Critic loss influence the shared parameters. This can create conflicting updates if the objectives are not well-aligned or if the loss magnitudes differ significantly. Careful tuning of learning rates or loss weights might be required.
• Compromised Representation: The shared layers might need to make compromises to serve both the Actor and Critic heads, potentially leading to suboptimal representations for one or both.

In practice, sharing parameters is common and often effective, especially when starting with standard implementations like A2C or A3C.

Actor Loss Function

The Actor's goal is to adjust its policy parameters $\theta$ to maximize the expected return. This is typically achieved by performing gradient ascent on an objective function $J(\theta)$. Equivalently, we minimize the negative of this objective, yielding the Actor loss $L_{actor}(\theta)$.

Using the policy gradient theorem and the Critic's estimate to reduce variance, the gradient update is typically based on the form:

$$\nabla_\theta J(\theta) \approx \mathbb{E}_t [ \nabla_\theta \log \pi_\theta(a_t|s_t) A_t ]$$

Here, $A_t$ represents the advantage of taking action $a_t$ in state $s_t$. This advantage is usually estimated using the Critic. A common estimate is based on the TD error $\delta_t$:

$$A_t \approx \delta_t = R_{t+1} + \gamma V_\phi(s_{t+1}) - V_\phi(s_t)$$

Where $V_\phi(s)$ is the state-value estimate provided by the Critic network with parameters $\phi$, $R_{t+1}$ is the reward received after taking action $a_t$, and $\gamma$ is the discount factor. Note that when calculating the gradient $\nabla_\theta$, the advantage term $A_t$ (and thus the Critic's output $V_\phi$) is treated as a constant baseline, not dependent on $\theta$.

The Actor loss function to be minimized is then:

$$L_{actor}(\theta) = - \mathbb{E}_t [ \log \pi_\theta(a_t|s_t) A_t ]$$

This loss encourages actions that lead to a higher-than-expected advantage (positive $A_t$) and discourages actions leading to a lower-than-expected advantage (negative $A_t$).

Critic Loss Function

The Critic's objective is to learn an accurate estimate of the state-value function $V(s)$ (or sometimes the action-value function $Q(s, a)$). It is trained similarly to value functions in DQN or other value-based methods, typically using Temporal Difference (TD) learning.

The Critic minimizes the difference between its current estimate $V_\phi(s_t)$ and a target value. A common target is the TD target, $Y_t = R_{t+1} + \gamma V_\phi(s_{t+1})$, assuming the episode hasn't terminated. The Critic loss $L_{critic}(\phi)$ is often the Mean Squared Error (MSE) between the estimate and the target:

$$L_{critic}(\phi) = \mathbb{E}_t [ (Y_t - V_\phi(s_t))^2 ] = \mathbb{E}_t [ (R_{t+1} + \gamma V_\phi(s_{t+1}) - V_\phi(s_t))^2 ]$$

When calculating the gradient $\nabla_\phi$, the target $Y_t$ (including the $V_\phi(s_{t+1})$ term if bootstrapping) is often treated as a fixed target, similar to how target networks work in DQN. This helps stabilize the Critic's training. In practice, the gradient is only computed with respect to $V_\phi(s_t)$.

Entropy Regularization

Policy gradient methods, including Actor-Critic, can sometimes converge prematurely to a suboptimal deterministic policy. To encourage exploration and prevent the policy from becoming too confident too quickly, entropy regularization is often added.

The entropy $H(\pi_\theta(\cdot|s_t))$ measures the randomness or uncertainty of the policy's action distribution at state $s_t$. By adding an entropy bonus to the objective function (or subtracting an entropy penalty from the loss), we encourage the policy to maintain a degree of randomness.

The modified Actor loss becomes:

$$L_{actor+entropy}(\theta) = - \mathbb{E}_t [ \log \pi_\theta(a_t|s_t) A_t + \beta H(\pi_\theta(\cdot|s_t)) ]$$

Here, $\beta$ is a hyperparameter controlling the strength of the entropy regularization. A higher $\beta$ encourages more exploration. The calculation of entropy depends on the policy distribution:

• For a Categorical distribution (discrete actions): $H = - \sum_a \pi_\theta(a|s_t) \log \pi_\theta(a|s_t)$.
• For a Gaussian distribution (continuous actions): It depends on the standard deviation(s).

Adding entropy regularization is a standard technique in modern Actor-Critic implementations like A2C and A3C.

Synchronous vs. Asynchronous Updates

The original A3C (Asynchronous Advantage Actor-Critic) used multiple worker threads, each interacting with its own copy of the environment. These workers computed gradients independently and applied them asynchronously to a shared global network. This asynchrony helped decorrelate the data used for updates, improving stability.

However, A2C (Advantage Actor-Critic) emerged as a popular alternative. A2C uses multiple workers in parallel to collect experience, but it performs synchronous updates. A central controller gathers experience batches from all workers, computes the gradients based on the aggregated batch, and updates the network parameters in one step. All workers then receive the updated parameters.

A2C often achieves similar or better performance than A3C, tends to utilize hardware like GPUs more efficiently due to batch processing, and can be simpler to implement. Many modern implementations favor the synchronous A2C approach, often using vectorized environments to manage the parallel simulations efficiently.

Handling State and Action Spaces

The network architecture must be adapted to the specific environment:

• State Input: If the state is represented by raw pixels (e.g., from a game screen), convolutional neural networks (CNNs) are typically used as the initial layers (often shared between Actor and Critic). If the state is a feature vector, standard multi-layer perceptrons (MLPs) are appropriate.
• Action Output (Actor):
  • Discrete Actions: The Actor network outputs logits (raw scores) for each possible action. These logits parameterize a Categorical distribution, from which an action is sampled.
  • Continuous Actions: The Actor network outputs parameters defining a continuous probability distribution, most commonly a Gaussian distribution. For instance, it might output the mean $\mu$ and standard deviation $\sigma$ for each action dimension. An action is then sampled from $N(\mu, \sigma^2)$.
• Value Output (Critic): The Critic network typically outputs a single scalar value representing the estimated state value $V(s)$.

These implementation choices, network structure, loss functions, regularization, and update strategy, are interdependent and require careful consideration and tuning to achieve effective learning in Actor-Critic agents. Experimentation with hyperparameters like learning rates (potentially separate ones for Actor and Critic), the entropy coefficient $\beta$, and the discount factor $\gamma$ is usually necessary.