Deep Deterministic Policy Gradient (DDPG)

While standard Actor-Critic methods provide a way to reduce variance compared to REINFORCE, applying them directly to environments with continuous action spaces presents challenges. Algorithms like A2C often rely on sampling actions from a stochastic policy $\pi(a|s)$ and computing expectations, which becomes complex when the action space is infinite.

Deep Deterministic Policy Gradient (DDPG) offers an elegant solution by adapting ideas from Deep Q-Networks (DQN) into an off-policy actor-critic framework specifically designed for continuous control. It was introduced by Lillicrap et al. in 2015 and has been influential in tackling problems like robotic manipulation and autonomous driving simulators.

From Stochastic to Deterministic Policies

The core modification in DDPG is the use of a deterministic policy, denoted as $\mu(s; \theta^\mu)$. Instead of outputting a probability distribution over actions, the actor network directly maps a state $s$ to a specific action $a$:

$$a = \mu(s; \theta^\mu)$$

Here, $\theta^\mu$ represents the parameters of the actor network. This shift simplifies the policy gradient computation significantly. Recall the standard policy gradient theorem involves an expectation over actions sampled from the policy. For a deterministic policy, the expectation disappears, and the gradient of the objective function $J(\theta^\mu)$ (expected return) can be derived using the chain rule, leveraging the critic's evaluation of the actor's chosen action.

The Actor and Critic Networks

DDPG maintains two primary neural networks, plus target versions of each:

1. Actor Network ($\mu(s; \theta^\mu)$): Takes the state $s$ as input and outputs a specific continuous action $a$. Its goal is to learn the optimal policy by maximizing the expected future reward.
2. Critic Network ($Q(s, a; \theta^Q)$): Takes both the state $s$ and an action $a$ as input and outputs the corresponding Q-value (expected return starting from $s$, taking action $a$, and following policy $\mu$ thereafter). Its goal is to accurately estimate the action-value function for the current actor policy.

Here's a visual representation of the DDPG architecture:

The DDPG agent interacts with the environment, storing experiences in a replay buffer. Updates involve sampling from the buffer, calculating target values using target networks, and adjusting actor and critic parameters based on the TD error and the policy gradient, respectively. Target networks are updated slowly towards the main networks.

Training the Critic

The critic network $Q(s, a; \theta^Q)$ is trained similarly to the Q-network in DQN. It aims to minimize the Mean Squared Bellman Error (MSBE). We sample a mini-batch of transitions $(s_i, a_i, r_i, s'_i)$ from the replay buffer $D$. For each transition, we compute the target value $y_i$:

$$y_i = r_i + \gamma Q'(s'_{i}, \mu'(s'_{i}; \theta^{\mu'}); \theta^{Q'})$$

Note that the target value calculation uses the target actor network $\mu'$ to select the next action $a'_{i} = \mu'(s'_{i}; \theta^{\mu'})$ and the target critic network $Q'$ to evaluate the value of that next state-action pair. This use of target networks ($Q'$ and $\mu'$) decouples the target value from the network being actively trained ($Q$), significantly improving stability, just like in DQN.

The critic's loss function is then the mean squared difference between the target values $y_i$ and the critic's current estimate $Q(s_i, a_i; \theta^Q)$:

$$L(\theta^Q) = \frac{1}{N} \sum_i (y_i - Q(s_i, a_i; \theta^Q))^2$$

This loss is minimized using gradient descent with respect to the critic parameters $\theta^Q$.

Training the Actor

The actor network $\mu(s; \theta^\mu)$ is updated using the deterministic policy gradient. The objective is to adjust the actor's parameters $\theta^\mu$ to produce actions that maximize the expected Q-value according to the current critic. The gradient of the actor's objective function $J(\theta^\mu)$ is approximated using the mini-batch from the replay buffer:

$$\nabla_{\theta^\mu} J(\theta^\mu) \approx \frac{1}{N} \sum_i \nabla_a Q(s_i, a; \theta^Q) |_{s=s_i, a=\mu(s_i; \theta^\mu)} \nabla_{\theta^\mu} \mu(s_i; \theta^\mu)$$

This looks complicated, but intuitively it means:

1. For each state $s_i$ in the batch, find the action $a = \mu(s_i; \theta^\mu)$ that the current actor policy would output.
2. Ask the critic how the Q-value $Q(s_i, a; \theta^Q)$ changes if we slightly change the action $a$ (this is $\nabla_a Q$).
3. Calculate how a change in the actor's parameters $\theta^\mu$ affects the output action $a$ (this is $\nabla_{\theta^\mu} \mu$).
4. Combine these gradients using the chain rule to determine how changing $\theta^\mu$ will affect the Q-value.
5. Update $\theta^\mu$ using gradient ascent (since we want to maximize $Q$).

This update uses the critic $Q$ as an evaluator, guiding the actor towards better actions without needing to sample actions and estimate expectations directly, making it suitable for continuous spaces.

Off-Policy Learning: Experience Replay and Target Networks

DDPG is an off-policy algorithm. Like DQN, it uses two mechanisms borrowed from value-based methods to improve stability and sample efficiency:

1. Experience Replay: Transitions $(s_t, a_t, r_t, s_{t+1})$ are stored in a large replay buffer $D$. During training, mini-batches are randomly sampled from $D$. This breaks the temporal correlations between consecutive experiences, leading to more stable and efficient learning as the network learns from a diverse set of past experiences.
2. Target Networks: DDPG maintains separate target networks, $\mu'(s; \theta^{\mu'})$ and $Q'(s, a; \theta^{Q'})$, which are slowly updated copies of the main actor and critic networks. Typically, a "soft" update rule called Polyak averaging is used after each main network update: $$\theta' \leftarrow \tau \theta + (1-\tau) \theta'$$ where $\tau \ll 1$ (e.g., $\tau = 0.001$ or $0.005$) is a hyperparameter controlling the update speed. These stable target networks provide consistent targets for the critic TD error calculation and stabilize the overall learning process.

Exploration in Deterministic Policies

Since the policy $\mu(s; \theta^\mu)$ is deterministic, it will always output the same action for a given state during training if left alone. This prevents exploration. To ensure the agent explores the environment sufficiently, DDPG adds noise to the actor's output action during training only:

$$a_t = \mu(s_t; \theta^\mu) + \mathcal{N}_t$$

The noise process $\mathcal{N}_t$ can be simple Gaussian noise, but the original DDPG paper used an Ornstein-Uhlenbeck (OU) process, which generates temporally correlated noise. This correlated noise can be helpful for exploration in physical control tasks where momentum is a factor. However, simpler uncorrelated Gaussian noise often works well too and is easier to implement. The scale of the noise is typically decayed over the course of training. During evaluation or deployment, this noise is turned off, and the agent acts purely based on $a_t = \mu(s_t; \theta^\mu)$.

DDPG Summary

DDPG combines the actor-critic structure with insights from DQN (replay buffer, target networks) to create an off-policy algorithm effective for continuous action spaces.

Strengths:

• Works directly with continuous action spaces.
• Off-policy nature allows for sample-efficient learning using a replay buffer.
• Relatively simpler concept compared to trust-region methods like TRPO.

Weaknesses:

• Like DQN, can be sensitive to hyperparameters (learning rates, target update rate $\tau$, noise parameters, network architecture).
• The critic can still suffer from Q-value overestimation, although target networks help mitigate this. Subsequent algorithms like TD3 (Twin Delayed DDPG) specifically address this limitation.
• Performance can sometimes be brittle depending on the environment and tuning.

Despite its weaknesses, DDPG represented a significant step forward for deep reinforcement learning in continuous domains and provides a foundation for understanding more recent algorithms like TD3 and SAC.