Actor-Critic Methods Overview

While policy gradient methods like REINFORCE allow us to optimize policies directly, they often suffer from high variance in their gradient estimates. This is because they typically rely on the Monte Carlo return $G_t$, the total accumulated reward from time step $t$ until the end of the episode. A single high or low reward late in an episode can significantly swing the estimated value of actions taken much earlier, making the learning process noisy and slow.

Actor-Critic methods offer a way to mitigate this variance while retaining the benefits of policy gradients. They represent a hybrid approach, combining elements from both policy-based methods (like REINFORCE) and value-based methods (like Q-learning or SARSA).

At its core, an Actor-Critic architecture consists of two distinct components, often implemented as separate neural networks or function approximators:

1. The Actor: This component is responsible for learning and executing the policy. It takes the current state $s$ as input and outputs a probability distribution over actions (for discrete action spaces) or the parameters of a distribution (for continuous action spaces). The Actor is essentially the parameterized policy $\pi_\theta(a|s)$ that we want to optimize. Its goal is to learn the parameters $\theta$ that maximize expected return.

2. The Critic: This component estimates a value function, typically either the state-value function $V(s)$ or the action-value function $Q(s,a)$. It takes state information (and possibly action information) as input and outputs an estimate of the value of being in that state or taking that action in that state. The Critic's role is to evaluate the actions taken by the Actor, providing feedback on their quality. It learns its own set of parameters, often denoted $w$, for the value function approximator (e.g., $V_w(s)$ or $Q_w(s,a)$).

How They Work Together

The Actor and Critic learn concurrently, interacting with the environment and each other. A typical interaction cycle looks like this:

1. Actor Acts: Given the current state $S_t$, the Actor selects an action $A_t$ according to its current policy $\pi_\theta(A_t|S_t)$.
2. Environment Responds: The agent performs action $A_t$ in the environment, receiving the next state $S_{t+1}$ and reward $R_{t+1}$.
3. Critic Evaluates: The Critic uses the observed transition $(S_t, A_t, R_{t+1}, S_{t+1})$ to evaluate the Actor's action. A common way to do this is by calculating the Temporal Difference (TD) error, which measures the difference between the estimated value and the observed reward plus the discounted value of the next state:
   • If using a state-value function $V_w(s)$: $$\delta_t = R_{t+1} + \gamma V_w(S_{t+1}) - V_w(S_t)$$
   • If using an action-value function $Q_w(s, a)$: $$\delta_t = R_{t+1} + \gamma Q_w(S_{t+1}, A_{t+1}) - Q_w(S_t, A_t)$$ (Where $A_{t+1}$ is chosen according to the Actor's policy or derived from the Critic's Q-values, depending on the specific algorithm variant). The TD error $\delta_t$ provides a signal indicating whether the outcome of taking action $A_t$ in state $S_t$ was better ($\delta_t > 0$) or worse ($\delta_t < 0$) than expected according to the Critic's current value estimate.
4. Updates:
   • Critic Update: The Critic updates its parameters $w$ to improve its value estimates, typically using a TD learning rule to minimize the TD error. For $V_w(S_t)$, the update might look like $w \leftarrow w + \alpha_C \delta_t \nabla_w V_w(S_t)$, where $\alpha_C$ is the Critic's learning rate.
   • Actor Update: The Actor updates its policy parameters $\theta$ based on the Critic's evaluation. The update direction aims to increase the probability of actions that led to better-than-expected outcomes (positive TD error) and decrease the probability of actions that led to worse-than-expected outcomes (negative TD error). The update often takes a form related to the policy gradient theorem, but using the TD error $\delta_t$ instead of the full return $G_t$: $$\theta \leftarrow \theta + \alpha_A \delta_t \nabla_\theta \log \pi_\theta(A_t|S_t)$$ Here, $\alpha_A$ is the Actor's learning rate. Notice how the Critic's evaluation $\delta_t$ directly modulates the policy update.

Basic interaction loop in an Actor-Critic architecture. The Actor chooses actions, the Critic evaluates them using information from the environment, and both components update their parameters based on this evaluation.

Advantages over REINFORCE

The primary advantage of Actor-Critic methods over basic policy gradient methods like REINFORCE is reduced variance. By using the TD error $\delta_t$ (which depends only on the next reward $R_{t+1}$ and the estimated value of the next state $S_{t+1}$) instead of the full Monte Carlo return $G_t$, the updates become less noisy. The Critic provides a more stable, learned baseline that adapts during training. This often leads to faster and more stable convergence.

Furthermore, because they rely on TD updates, Actor-Critic methods can learn online (updating after each step) and can be applied more naturally to continuing tasks (tasks without a defined end).

Actor-Critic methods form the foundation for many advanced RL algorithms. While this overview covers the basic concept, numerous variations exist, differing in how the value function is estimated, how the updates are performed, and how the Actor and Critic networks are structured (e.g., Advantage Actor-Critic (A2C), Asynchronous Advantage Actor-Critic (A3C), Deep Deterministic Policy Gradient (DDPG)). However, the fundamental idea remains the same: use a Critic to learn a value function that provides low-variance feedback to guide the Actor's policy updates.