Return: Cumulative Future Rewards

Reinforcement Learning involves an agent interacting with its environment over a sequence of time steps. At each step $t$, the agent observes a state $S_t$, chooses an action $A_t$, receives a reward $R_{t+1}$, and transitions to a new state $S_{t+1}$. While rewards provide an immediate feedback mechanism, the agent's objective isn't merely to maximize the immediate reward $R_{t+1}$. Intelligent behavior often requires considering the long-term consequences of actions. An action might yield a small immediate reward but lead to states offering much larger future rewards. Conversely, a large immediate reward might lead to unfavorable future states.

To capture this of long-term cumulative reward, we introduce the Return. The return, denoted $G_t$, is the total reward an agent expects to accumulate starting from time step $t$ onwards. The precise way we calculate the return depends on whether the task is episodic or continuing.

Return in Episodic Tasks

Episodic tasks are those that have a natural ending point. Think of games like chess or Pac-Man, where each game eventually terminates. In these tasks, the sequence of states, actions, and rewards forms an episode: $S_0, A_0, R_1, S_1, A_1, R_2, ..., S_{T-1}, A_{T-1}, R_T, S_T$, where $S_T$ is a special terminal state.

For an episodic task, the return $G_t$ starting from time step $t$ is simply the sum of all rewards received from step $t+1$ until the end of the episode at step $T$:

$$G_t = R_{t+1} + R_{t+2} + R_{t+3} + \dots + R_T$$

This can be written more compactly as:

$$G_t = \sum_{k=0}^{T-t-1} R_{t+k+1}$$

Here, $T$ is the final time step of the episode. The goal of the agent in an episodic task is to choose actions that maximize the expected value of this finite sum of future rewards for each episode.

Return in Continuing Tasks and the Need for Discounting

Continuing tasks, on the other hand, do not have a terminal state and could potentially go on forever. Examples include controlling a robot performing an ongoing maintenance task or managing an investment portfolio. If we simply summed the rewards as we did for episodic tasks, the return $G_t$ could easily become infinite ($T \to \infty$), making it difficult to compare different strategies. How can we meaningfully define a cumulative reward that doesn't diverge?

This is where the concept of discounting becomes essential. We introduce a discount factor, denoted by the Greek letter gamma ($\gamma$), where $0 \le \gamma \le 1$. The idea is to give less weight to rewards received further in the future compared to immediate rewards.

The discounted return $G_t$ is defined as the sum of future rewards, where each reward $R_{t+k+1}$ is multiplied by $\gamma^k$:

$$G_t = R_{t+1} + \gamma R_{t+2} + \gamma^2 R_{t+3} + \dots = \sum_{k=0}^{\infty} \gamma^k R_{t+k+1}$$

Why does this help?

1. Mathematical Convergence: If the reward signal is bounded (which is often the case) and $\gamma < 1$, this infinite sum converges to a finite value. This allows us to compare returns even for infinite sequences.
2. Behavioral Preference: Discounting often makes intuitive sense. A reward received sooner is generally preferable to the same reward received later. The factor $\gamma$ controls this preference:
   • If $\gamma$ is close to 0, the agent becomes "myopic," focusing heavily on immediate rewards.
   • If $\gamma$ is close to 1, the agent becomes "far-sighted," taking future rewards into account more strongly.

A Unified View

Interestingly, the discounted return formula $G_t = \sum_{k=0}^{\infty} \gamma^k R_{t+k+1}$ can be seen as a general case. For episodic tasks, we can consider them to end by transitioning to an absorbing terminal state that only yields rewards of 0 from then on. The formula still applies, and if $\gamma = 1$, it simplifies back to the undiscounted sum $G_t = \sum_{k=0}^{T-t-1} R_{t+k+1}$. However, using $\gamma < 1$ is common even in episodic tasks to encourage finding the terminal state faster or to ensure convergence in certain algorithms.

Therefore, the fundamental goal in most RL problems framed within MDPs is to find a policy (a way of choosing actions) that maximizes the expected discounted return $E[G_t]$ from each state $S_t$. This quantity, the expected return under a specific policy, is precisely what we will define as the value of a state or state-action pair in subsequent sections. Understanding the return $G_t$ is the first step towards evaluating how good different situations and actions are in the long run. We will explore the role of the discount factor $\gamma$ in more detail next.