On-Policy vs Off-Policy Learning

Learning from sampled episodes of experience, Monte Carlo methods raise a fundamental question: which policy is actually being learned? The answer to this question distinguishes between two main categories of RL algorithms: on-policy and off-policy learning.

Understanding this distinction is significant because it affects how we can gather experience and what we can learn from it. It influences algorithm design, stability, and flexibility, particularly when balancing exploration (trying new actions) and exploitation (using known good actions).

On-Policy Learning: Learning on the Job

In on-policy methods, the agent learns about the value function or policy it is currently following. Think of it as learning "on the job." The policy used to generate the interaction data (the episodes) is the same policy that the agent is trying to evaluate and improve.

Let's say the agent is following a policy $\pi$. It interacts with the environment using actions chosen according to $\pi$. It collects episodes like $(S_0, A_0, R_1, S_1, A_1, R_2, ..., S_T)$. The returns calculated from these episodes are then used to estimate $V^\pi(s)$ or $Q^\pi(s, a)$, the value functions for this specific policy $\pi$. Subsequently, $\pi$ is improved based on these updated value estimates, perhaps by making it more greedy with respect to the learned action values. The next batch of experience is then generated using this improved policy.

Characteristics of On-Policy Learning:

• Direct Experience: Learns directly from the consequences of the actions taken by the current policy.
• Policy Evaluation and Improvement: The policy being evaluated and improved is the same one generating the data.
• Exploration within the Policy: Exploration strategies (like $\epsilon$-greedy, where the agent takes a random action with probability $\epsilon$) must often be part of the policy $\pi$ itself to ensure sufficient coverage of the state-action space. This means the policy being learned is often inherently exploratory (e.g., learning the value of an $\epsilon$-greedy policy).
• Simplicity: Straightforward; the agent learns from what it does.

A typical example within Monte Carlo is On-Policy First-Visit MC Control, which we will detail shortly. It uses returns generated by an $\epsilon$-soft policy to improve that same $\epsilon$-soft policy.

Off-Policy Learning: Learning from Others' Experience

In off-policy methods, the agent learns about a target policy $\pi$ using data generated from following a different behavior policy $\mu$. Think of this as learning how to perform a task optimally ($\pi$) by observing someone else, possibly less skilled or more exploratory ($\mu$), perform it.

The target policy $\pi$ is usually the policy the agent ultimately wants to learn (e.g., the deterministic greedy policy that always chooses the action with the highest estimated value). The behavior policy $\mu$, however, is the one used to actually interact with the environment and generate episodes. $\mu$ might be more exploratory than $\pi$ to ensure it gathers data about a wider range of actions and states.

Why use Off-Policy Learning?

• Learn Optimal Policy While Exploring: The agent can behave using a highly exploratory policy ($\mu$) to gather diverse experiences, while simultaneously using that experience to learn about the optimal, deterministic policy ($\pi$).
• Learning from Historical Data: It allows learning from data generated by previous policies or even human demonstrations (where the behavior policy $\mu$ is the policy followed by the human or the old agent).
• Flexibility: Decouples data generation from the policy being learned.

The Challenge: Since the data comes from $\mu$, not $\pi$, the distribution of states visited and actions taken doesn't directly reflect what would happen under $\pi$. Actions preferred by $\pi$ might be rare in the data generated by $\mu$, and vice versa. Directly averaging returns from $\mu$'s episodes would yield estimates for $V^\mu$ or $Q^\mu$, not $V^\pi$ or $Q^\pi$.

The Solution: Off-policy methods need a way to correct for this mismatch. They often use techniques like Importance Sampling, which involves weighting the observed returns based on the relative probability of the experienced trajectory occurring under the target policy $\pi$ versus the behavior policy $\mu$. This adjusts the contribution of each episode to account for the difference in policies. We'll touch upon Importance Sampling in the context of Off-Policy MC later in this chapter.

Characteristics of Off-Policy Learning:

• Indirect Experience: Learns about one policy using data generated by another.
• Separate Target and Behavior Policies: Requires distinguishing between the policy being learned ($\pi$) and the policy generating data ($\mu$).
• Requires Correction: Needs mechanisms (like importance sampling) to account for the difference between $\pi$ and $\mu$.
• Potential for Higher Variance: Importance sampling can sometimes lead to high variance in the value estimates, especially if the policies $\pi$ and $\mu$ are very different.

Q-learning, which we will encounter in the next chapter on Temporal-Difference learning, is a well-known off-policy algorithm.

Diagram comparing On-Policy and Off-Policy learning flows. On-policy uses the same policy ($\pi$) for interaction and learning. Off-policy uses a behavior policy ($\mu$) for interaction to learn about a different target policy ($\pi$), requiring a correction step.

Summary Table

Feature	On-Policy Learning	Off-Policy Learning
Data Source	Policy being learned ($\pi$)	Separate behavior policy ($\mu$)
Target Policy	Policy being learned ($\pi$)	Often a different policy ($\pi$, e.g., greedy)
Exploration	Built into the learning policy ($\pi$)	Handled by behavior policy ($\mu$)
Challenge	Balancing exploration/exploitation	Correcting for policy mismatch (variance)
Flexibility	Lower (learns about what it does)	Higher (can learn optimal policy while exploring, use historical data)
Example	On-Policy MC, SARSA	Off-Policy MC, Q-Learning

In essence, the choice between on-policy and off-policy methods depends on the problem requirements. If you need to learn the value of the specific behavior strategy being used (including its exploration), on-policy is suitable. If you want to learn an optimal policy regardless of the exploration strategy used during data collection, or if you want to learn from data generated by a different agent or policy, off-policy methods provide the necessary mechanisms. As we proceed with Monte Carlo control, we'll see how this distinction plays out in algorithm design.