Introduction to Offline RL (Batch RL)

Reinforcement learning commonly involves agents continuously interacting with their environment, collecting data, and updating their strategies in a loop. A distinct, yet increasingly relevant, learning approach is Offline Reinforcement Learning, also commonly referred to as Batch Reinforcement Learning.

Why Learn Offline?

Imagine scenarios where deploying an exploratory RL agent directly into the real system is impractical, unsafe, or prohibitively expensive. Consider:

• Healthcare: Testing new treatment policies directly on patients is unethical. Learning from existing (anonymized) patient records is a more viable approach.
• Robotics: Allowing a physical robot to explore randomly during initial learning phases can lead to costly damage or unsafe situations. Learning from logged sensor data or demonstrations is preferred.
• Autonomous Driving: Exploration on public roads carries significant risks. Utilizing large logs of driving data collected by human drivers or existing systems offers a safer alternative.
• Recommender Systems/Online Advertising: Companies often possess massive logs of past user interactions. Learning improved recommendation or bidding strategies from this historical data, without interrupting the live system with potentially suboptimal exploratory actions, is highly desirable.

In these cases, and many others, we are presented with a fixed dataset of interactions collected beforehand, possibly by a different policy or even multiple policies. The objective is to leverage this static dataset to learn the best policy possible without any further interaction with the environment during the learning process. This is the core idea behind Offline RL.

The Offline RL Problem Setup

Formally, the setup for Offline RL is distinct from the online setting. We are given a static dataset $D$, which consists of a collection of transitions:

$$D = \{ (s_i, a_i, r_i, s'_i) \}_{i=1}^N$$

Here, $s_i$ is the state, $a_i$ is the action taken in that state, $r_i$ is the received reward, and $s'_i$ is the resulting next state. This dataset $D$ was collected using one or more behavior policies, collectively denoted as $\pi_b$. Importantly, we might not know the exact policy $\pi_b$ that generated the data. The main constraint is that our learning algorithm can only use the transitions present in $D$. It cannot query the environment for the outcome of new state-action pairs.

The goal is to use this dataset $D$ to learn a target policy $\pi$ that maximizes the expected cumulative reward when deployed in the actual environment.

The Offline Reinforcement Learning process: Data is collected from environment interactions using behavior policies, forming a static dataset. The Offline RL algorithm learns a new policy solely from this dataset, without further environment interaction.

Contrasting with Online and Standard Off-Policy Learning

It's important to distinguish Offline RL from related concepts:

1. Online RL: The agent continuously interacts with the environment, gathers experience (transitions), and updates its policy based on this incoming stream of data. Exploration is an active part of the learning loop. Examples include SARSA, A2C/A3C (when run online), and online Q-learning.
2. Off-Policy RL (Online Setting): The agent learns about a target policy $\pi$ (e.g., the greedy policy) while following a different behavior policy $\pi_b$ (e.g., epsilon-greedy) to collect data. Algorithms like DQN and DDPG are off-policy. However, in the standard online off-policy setting, the agent still interacts with the environment, collecting new transitions using $\pi_b$ and adding them to its replay buffer. The dataset grows and evolves over time.
3. Offline RL (Batch RL): This is a stricter form of off-policy learning. Like standard off-policy methods, it learns about a policy $\pi$ from data generated by $\pi_b$. However, the defining characteristic is that the dataset $D$ is fixed and collected entirely before learning begins. There is absolutely no further interaction with the environment during the training phase.

The Specter of Distributional Shift

Why is learning purely from a static dataset so challenging? The primary obstacle is distributional shift. The fixed dataset $D$ reflects the state-action visitation frequency induced by the behavior policy $\pi_b$. If our learning algorithm tries to evaluate or improve a target policy $\pi$ that behaves differently from $\pi_b$, $\pi$ might favor actions or lead to states that are poorly represented or entirely absent in $D$.

Consider training a Q-network using standard off-policy algorithms like DQN on the offline dataset $D$. The Bellman update involves terms like $\max_{a'} Q(s', a')$. If the state $s'$ is encountered in the dataset, but the action $a' = \arg\max_{a'} Q(s', a')$ was never taken by $\pi_b$ in state $s'$, the Q-value $Q(s', a')$ is estimated purely through function approximation extrapolation. Neural networks are notoriously unreliable when extrapolating outside the distribution of their training data. This can lead to wildly inaccurate Q-value estimates, often significant overestimation, causing the learning process to diverge or converge to a poor policy. This failure mode, stemming from querying the value of actions not present in the batch for specific states, is a direct consequence of distributional shift.

Addressing this challenge is the central theme of modern Offline RL research. The subsequent sections will examine techniques developed specifically to handle distributional shift, including methods for evaluating policies offline (Offline Policy Evaluation) and algorithms designed for robust offline policy optimization.