Actor-Critic Architecture Fundamentals

Standard policy gradient methods like REINFORCE update the policy parameters $\theta$ based on the sampled return $G_t$ from an entire episode. The gradient estimate often looks like $\nabla_\theta J(\theta) \approx \mathbb{E}[\nabla_\theta \log \pi_\theta(A_t|S_t) G_t]$. While unbiased, relying on the full Monte Carlo return $G_t$ introduces significant variance because the return depends on all subsequent actions and rewards in the trajectory. A single high or low reward can drastically swing the gradient estimate, slowing down or destabilizing learning.

Actor-Critic methods offer a compelling alternative structure to mitigate this high variance. Instead of one component learning the policy, we introduce two distinct components, typically implemented as separate function approximators (often neural networks):

1. The Actor: This component is responsible for learning and representing the policy. It takes the current state $s$ as input and outputs a probability distribution over actions (for stochastic policies) or a specific action (for deterministic policies). We denote the actor's policy as $\pi_\theta(a|s)$, parameterized by $\theta$. The actor's goal is to learn the optimal policy by adjusting $\theta$.

2. The Critic: This component learns a value function to evaluate the actions chosen by the actor or the states encountered. It takes state $s$ (and sometimes action $a$) as input and outputs an estimate of value. Common choices for the critic's function are the state-value function $V_\phi(s)$ or the action-value function $Q_\phi(s,a)$, parameterized by $\phi$. The critic's role is not to select actions but to provide feedback on how good the actor's current policy is.

How They Interact

The core idea is that the actor decides what to do, and the critic evaluates how well it was done. This evaluation then guides the actor's updates. Here's the typical flow:

1. Action Selection: The actor observes the current state $S_t$ and selects an action $A_t$ according to its policy $\pi_\theta(A_t|S_t)$.
2. Environment Interaction: The action $A_t$ is performed in the environment, leading to a reward $R_{t+1}$ and the next state $S_{t+1}$.
3. Critic Evaluation: The critic uses the transition information $(S_t, A_t, R_{t+1}, S_{t+1})$ to evaluate the actor's action or the resulting state. A common way is to compute the Temporal Difference (TD) error: $$\delta_t = R_{t+1} + \gamma V_\phi(S_{t+1}) - V_\phi(S_t)$$ This TD error measures the discrepancy between the critic's current estimate $V_\phi(S_t)$ and a potentially better estimate based on the immediate reward and the value of the next state (the TD target $R_{t+1} + \gamma V_\phi(S_{t+1})$). If the critic estimates $Q_\phi(s,a)$, the TD error might be $\delta_t = R_{t+1} + \gamma Q_\phi(S_{t+1}, A_{t+1}) - Q_\phi(S_t, A_t)$, where $A_{t+1}$ is selected by the actor in state $S_{t+1}$.
4. Critic Update: The critic updates its parameters $\phi$ to minimize this TD error, usually via gradient descent. The goal is to make its value estimates more accurate over time. For a state-value critic, the update might target minimizing $(\delta_t)^2$.
5. Actor Update: The actor updates its policy parameters $\theta$ in the direction suggested by the critic's evaluation. Instead of using the noisy Monte Carlo return $G_t$, the actor uses the critic's feedback, often the TD error $\delta_t$ or a related quantity like the advantage $A(S_t, A_t)$. The policy gradient update becomes something like: $$\nabla_\theta J(\theta) \approx \mathbb{E}[\nabla_\theta \log \pi_\theta(A_t|S_t) \delta_t] \quad \text{(or using Advantage)}$$ A positive TD error suggests the action $A_t$ led to a better-than-expected outcome, so the probability of selecting $A_t$ in state $S_t$ should be increased. A negative TD error suggests the opposite.

This interaction loop allows the actor and critic to improve concurrently. The critic learns to provide better evaluations, and the actor learns to produce better actions based on those evaluations.

Basic Actor-Critic interaction loop. The actor selects actions based on the state, the environment responds, and the critic evaluates the outcome, providing feedback used to update both the actor's policy and its own value estimates.

Benefits of the Actor-Critic Structure

Compared to REINFORCE, the primary advantage of the Actor-Critic framework is variance reduction. The TD error $\delta_t$ (or advantage) used for the actor update depends mainly on the immediate reward $R_{t+1}$ and the critic's estimate of the next state's value $V_\phi(S_{t+1})$. This estimate, while potentially biased (since $V_\phi$ is learned), is generally much less noisy than the full Monte Carlo return $G_t$, which accumulates noise over many time steps. Lower variance often leads to faster and more stable convergence.

Furthermore, Actor-Critic methods can learn online, updating the actor and critic after each step (or small batch of steps), unlike basic REINFORCE which typically requires waiting until the end of an episode to compute $G_t$.

Variants

The specific form of the critic's value function significantly impacts the algorithm:

• State-Value Critic ($V_\phi(s)$): The critic estimates the value of being in a state. The evaluation signal is often the TD error $\delta_t = R_{t+1} + \gamma V_\phi(S_{t+1}) - V_\phi(S_t)$. This $\delta_t$ directly approximates the advantage $A(S_t, A_t)$ under certain assumptions.
• Action-Value Critic ($Q_\phi(s,a)$): The critic estimates the value of taking action $a$ in state $s$. This is common in algorithms designed for continuous action spaces (like DDPG) or when a direct estimate of Q-values is needed. The update typically involves the TD error $\delta_t = R_{t+1} + \gamma Q_\phi(S_{t+1}, A_{t+1}) - Q_\phi(S_t, A_t)$.
• Advantage Function Critic: Some methods explicitly try to estimate the advantage function $A(s,a) = Q(s,a) - V(s)$. Using the advantage directly in the policy gradient update, $\nabla_\theta J(\theta) \approx \mathbb{E}[\nabla_\theta \log \pi_\theta(A_t|S_t) A(S_t, A_t)]$, is often beneficial because it provides a relative measure of how much better an action is compared to the average action from that state, further reducing variance. We will see sophisticated methods for estimating the advantage, like Generalized Advantage Estimation (GAE), later in this chapter.

In deep reinforcement learning, both the actor $\pi_\theta$ and the critic ($V_\phi$ or $Q_\phi$) are usually represented by neural networks. Their parameters, $\theta$ and $\phi$, are updated using gradient-based optimization techniques based on the principles outlined above.

This fundamental Actor-Critic architecture serves as the foundation for many powerful algorithms we will discuss next, including Advantage Actor-Critic (A2C/A3C), Deep Deterministic Policy Gradient (DDPG), Proximal Policy Optimization (PPO), and Soft Actor-Critic (SAC). These methods refine the basic structure by improving how the advantage is estimated, how the updates are performed for better stability, or how exploration is handled.