Challenge: Distributional Shift

The fundamental constraint of Offline Reinforcement Learning (RL) is that we cannot interact with the environment to gather new data. We are confined to a static dataset, $\mathcal{D} = \{(s_i, a_i, r_i, s'_i)\}_{i=1}^N$, collected possibly by a different policy (or mixture of policies), known as the behavior policy $\pi_b$. Our goal is to learn a new, potentially better, target policy $\pi$ (or evaluate a given one) using only $\mathcal{D}$. This is where distributional shift becomes a significant impediment.

Distributional shift refers to the mismatch between the distribution of state-action pairs encountered in the offline dataset, $d^{\pi_b}(s, a)$, and the distribution of state-action pairs that the learned policy $\pi$ would induce if deployed in the environment, $d^{\pi}(s, a)$.

Why is this mismatch a problem? Many standard RL algorithms, especially off-policy methods like Q-learning and its deep variants (DQN), rely on evaluating the expected return of taking actions in given states. Consider the Bellman update used in Q-learning:

$$Q(s, a) \leftarrow \mathbb{E}_{r, s' \sim \mathcal{D}}[r + \gamma \max_{a'} Q(s', a')]$$

Or, more accurately in the offline, function approximation setting, we often minimize a loss like the Mean Squared Bellman Error (MSBE) over the dataset:

$$L(\theta) = \mathbb{E}_{(s, a, r, s') \sim \mathcal{D}} \left[ \left( r + \gamma \max_{a'} Q_{\bar{\theta}}(s', a') - Q_{\theta}(s, a) \right)^2 \right]$$

Here, $Q_{\theta}$ is our parameterized Q-function and $Q_{\bar{\theta}}$ is a target network. The issue arises within the term $\max_{a'} Q_{\bar{\theta}}(s', a')$. The policy $\pi$ derived from $Q_{\theta}$ (e.g., $\pi(s) = \arg\max_a Q_{\theta}(s, a)$) might suggest taking actions $a'$ in states $s'$ that were rarely or never visited together under the behavior policy $\pi_b$. That is, the probability $d^{\pi_b}(s', a')$ might be very low or zero for the action $a'$ that maximizes the learned $Q_{\bar{\theta}}(s', \cdot)$.

In online and standard off-policy RL, if the agent starts favoring a new action $a'$ in state $s'$, the exploration mechanism (like epsilon-greedy) gives it a chance to actually try $(s', a')$ in the real environment and receive feedback (reward $r'$ and next state $s''$). This grounds the Q-value estimate in actual experience. In the offline setting, this is impossible. The algorithm can only learn from the transitions present in $\mathcal{D}$.

Consequences of Distributional Shift

1. Extrapolation Errors: Neural networks and other function approximators are effective at interpolating within the data distribution they were trained on. However, they often perform poorly when asked to extrapolate to inputs far outside that distribution. Evaluating $Q(s', a')$ for an out-of-distribution (OOD) action $a'$ is precisely such an extrapolation task. The resulting Q-value can be arbitrarily inaccurate.

2. Overestimation Bias: The $\max$ operator in the Q-learning target $\max_{a'} Q(s', a')$ tends to pick actions with the highest Q-values. If some of these high values are due to extrapolation errors on OOD actions, the target value itself becomes erroneously high. This leads to optimistic bias for state-action pairs $(s, a)$ that transition to states $s'$ where OOD actions look good. This bias can propagate through the learning process via bootstrapping, systematically inflating value estimates across many states.

3. Policy Degradation: Learning a policy based on these erroneously optimistic Q-values is problematic. The policy might learn to favor actions that appear good according to the flawed $Q$-function but would actually perform poorly if executed in the real environment because their values were overestimated due to distributional shift.

Representation of distributional shift. The offline dataset contains state-action pairs (blue nodes) visited by the behavior policy $\pi_b$. Standard off-policy algorithms might need to estimate Q-values for actions (like $a_2$ in state $s_1$, red node) that are absent or rare in the dataset during the update process or when deriving the learned policy $\pi$. This reliance on out-of-distribution estimates introduces significant risks.

Effectively managing distributional shift is therefore a central theme in offline RL. Algorithms must be designed to either prevent the policy from querying OOD actions or correct the value estimates for such actions, ensuring that learning remains grounded in the available data. Subsequent sections will explore specific algorithmic techniques developed to address this challenge.