Basic policy gradient methods, like REINFORCE, often suffer from high variance in their gradient estimates, leading to slow or unstable learning. This chapter introduces Actor-Critic methods, a family of algorithms designed to address this limitation. The core idea is to maintain two components: an actor that learns the policy $\pi(a|s)$ and a critic that learns a value function (like $V(s)$ or $Q(s,a)$) to evaluate the actor's actions and provide lower-variance gradient signals.

You will study several key advancements building on this framework:

• Variance Reduction: Techniques like using baselines and Advantage Actor-Critic (A2C/A3C) to stabilize policy updates.
• Advantage Estimation: Understanding Generalized Advantage Estimation (GAE) for balancing bias and variance.
• Continuous Control: Algorithms like Deep Deterministic Policy Gradient (DDPG) adapted for continuous action spaces.
• Policy Optimization Stability: Methods ensuring more reliable policy improvements, specifically Trust Region Policy Optimization (TRPO) and Proximal Policy Optimization (PPO).
• Maximum Entropy RL: Soft Actor-Critic (SAC), an off-policy method incorporating entropy maximization for better exploration and stability.

By the end of this chapter, you will understand the theory behind these advanced algorithms and be prepared to implement them for solving more complex reinforcement learning problems.