Conservative Q-Learning (CQL)

While policy constraint methods like BCQ directly restrict the learned policy to actions resembling those in the offline dataset, Conservative Q-Learning (CQL) takes a different approach. Instead of modifying the policy search space, CQL modifies the Q-function learning objective itself to combat the overestimation of values for out-of-distribution (OOD) actions, a primary consequence of distributional shift.

The fundamental problem arises when standard Q-learning updates, like those in DQN or SAC, use the maximum Q-value over actions in the Bellman target: $y = r + \gamma \max_{a'} Q_{\bar{\theta}}(s', a')$. If the Q-function $Q_{\bar{\theta}}$ incorrectly assigns high values to OOD actions $a'$ (actions unlikely or unseen under the behavior policy $\pi_b$ that generated the dataset $\mathcal{D}$), these errors propagate through the learning process, leading to a suboptimal final policy.

The CQL Principle: Penalizing OOD Action Values

CQL introduces a regularization term into the standard Q-learning loss function. This regularizer encourages the Q-function $Q_\theta(s, a)$ to assign low values to OOD actions while ensuring that the Q-values for actions actually present in the dataset $\mathcal{D}$ remain accurate or are pushed higher by the Bellman updates. This results in a "conservative" estimate of the Q-values, preventing the agent from exploiting potentially overestimated values of unseen actions.

The CQL Objective Function

The CQL objective typically augments a standard TD-based loss, $L_{TD}(\theta)$, with a state-action value regularizer, $R_{CQL}(\theta)$. The combined objective is:

$L_{CQL}(\theta) = \alpha R_{CQL}(\theta) + L_{TD}(\theta)$

Here, $\alpha \ge 0$ is a hyperparameter that weights the conservatism penalty. $L_{TD}(\theta)$ could be the mean squared Bellman error from DQN or the soft Bellman residual from SAC.

The core component is the regularizer $R_{CQL}(\theta)$. A common form aims to minimize the Q-values under some proposal distribution $\mu(a|s)$ while maximizing the Q-values for actions sampled from the dataset $\mathcal{D}$:

$R_{CQL}(\theta) = \mathbb{E}_{s \sim \mathcal{D}} \left[ \mathbb{E}_{a \sim \mu(a|s)} [Q_\theta(s, a)] - \mathbb{E}_{a \sim \pi_b(a|s)} [Q_\theta(s, a)] \right]$

Alternatively, using a log-sum-exp formulation (which implicitly covers all actions), the regularizer can be written as:

$R_{CQL}(\theta) = \mathbb{E}_{s \sim \mathcal{D}} \left[ \log \sum_a \exp(Q_\theta(s,a)) - \mathbb{E}_{a \sim \pi_b(a|s)}[Q_\theta(s,a)] \right]$

In practice, $\mathbb{E}_{a \sim \pi_b(a|s)}[Q_\theta(s,a)]$ is approximated using the specific action $a$ from the transition $(s, a, r, s')$ sampled from the dataset $\mathcal{D}$. The expectation $\mathbb{E}_{a \sim \mu(a|s)}$ or the log-sum-exp term requires sampling or evaluating Q-values for multiple actions at state $s$. These actions might be sampled uniformly or from the agent's current policy.

Intuition Behind the Regularizer

Let's break down the regularizer's effect:

1. Minimize Q-values for OOD actions: The first term (e.g., $\mathbb{E}_{a \sim \mu(a|s)} [Q_\theta(s, a)]$ or $\log \sum_a \exp(Q_\theta(s,a))$) effectively pushes down the estimated Q-values for actions that are not strongly represented in the dataset for state $s$. The log-sum-exp acts like a "soft maximum", penalizing states where any action has a high Q-value unless counteracted by the second term.
2. Support Q-values for In-Distribution actions: The second term (e.g., $- \mathbb{E}_{a \sim \pi_b(a|s)} [Q_\theta(s, a)]$) pushes up the Q-values specifically for the actions $a$ that were observed in the dataset $\mathcal{D}$ paired with state $s$.

Minimizing the overall $R_{CQL}(\theta)$ term forces the learned Q-function $Q_\theta(s, a)$ to have values for actions seen in the data that are higher than (or bounded below by) the values of actions not seen in the data. The TD loss $L_{TD}(\theta)$ simultaneously ensures these in-distribution Q-values are consistent with the Bellman equation.

Standard Q-learning might overestimate values for out-of-distribution (OOD) actions. CQL adds a penalty that pushes down the Q-values of OOD actions while ensuring the values for actions seen in the dataset remain grounded by the TD updates.

Advantages of Conservative Q-Learning

• Direct Mitigation of Overestimation: CQL directly targets the Q-function overestimation problem for OOD actions, which is a root cause of instability in offline RL.
• Theoretical Backing: Under certain conditions, CQL provides guarantees that the learned Q-function is a lower bound on the true Q-function, promoting conservative decision-making.
• Flexibility: CQL can be readily integrated with various Q-learning frameworks, including DQN and actor-critic methods like SAC. When used with actor-critic, the actor is trained to maximize the learned conservative Q-function.
• Strong Empirical Performance: CQL has demonstrated state-of-the-art or competitive results across numerous offline RL benchmark tasks.

Implementation Approaches

• Choice of Regularizer: Different forms of the CQL regularizer exist, tailored for discrete or continuous action spaces. The specific implementation details matter for performance.
• Hyperparameter $\alpha$: The conservatism weight $\alpha$ is significant. A small $\alpha$ may not sufficiently penalize OOD actions, while a very large $\alpha$ might overly suppress Q-values, potentially hindering learning of the optimal policy within the data distribution. Tuning $\alpha$ is often necessary.
• Action Sampling: For the OOD action penalization term (e.g., $\mathbb{E}_{a \sim \mu(a|s)} [Q_\theta(s, a)]$), actions need to be sampled from a suitable distribution $\mu$. Common choices include sampling uniformly at random or sampling from the current learned policy.

Compared to policy constraint methods, CQL offers a different philosophy. It allows the policy to potentially represent any action but relies on the learned conservative Q-values to guide the policy towards actions supported by the data. This value-based regularization provides a powerful alternative for tackling the challenges inherent in learning from fixed datasets.