Discounting Future Rewards

In the previous section, we defined the 'Return', $G_t$, as the total reward an agent expects to accumulate starting from time step $t$. For tasks that have a definite end (episodic tasks), this is simply the sum of all rewards from $t+1$ until the episode terminates.

However, many interesting RL problems involve ongoing interactions without a natural endpoint (continuing tasks). Imagine an agent managing a power grid or a trading bot operating continuously. If we simply sum rewards indefinitely, the Return $G_t$ could easily become infinite. Comparing infinite returns doesn't help us determine which policy is better. Furthermore, even in episodic tasks, should a reward received 100 steps into the future be valued the same as a reward received right now?

This brings us to the concept of discounting. We introduce a parameter, the discount factor, denoted by the Greek letter gamma ($\gamma$), to systematically reduce the value of future rewards.

The discount factor $\gamma$ is a number between 0 and 1 ($0 \le \gamma \le 1$). The return $G_t$ is now defined as the sum of discounted future rewards:

$$G_t = R_{t+1} + \gamma R_{t+2} + \gamma^2 R_{t+3} + \gamma^3 R_{t+4} + \dots = \sum_{k=0}^{\infty} \gamma^k R_{t+k+1}$$

Let's break down this definition:

• The reward received one step later, $R_{t+1}$, is added directly (multiplied by $\gamma^0 = 1$).
• The reward received two steps later, $R_{t+2}$, is multiplied by $\gamma$.
• The reward received three steps later, $R_{t+3}$, is multiplied by $\gamma^2$.
• And so on; the reward received $k+1$ steps later, $R_{t+k+1}$, is multiplied by $\gamma^k$.

Since $\gamma$ is less than or equal to 1, $\gamma^k$ gets smaller as $k$ increases. This means rewards received further in the future contribute less to the total discounted return $G_t$.

Why Discount?

Discounting serves several important purposes:

1. Mathematical Convenience: In continuing tasks, discounting ensures that the infinite sum defining $G_t$ converges to a finite value, provided the rewards are bounded. If the reward at any step is guaranteed to be no larger than some value $R_{max}$, then the return $G_t$ is guaranteed to be less than or equal to $\frac{R_{max}}{1-\gamma}$ (using the formula for the sum of an infinite geometric series). This makes comparing different policies mathematically tractable.
2. Modeling Preference for Near-Term Rewards: Often, receiving a reward sooner is preferable to receiving the same reward later. Think about financial interest rates or biological imperatives for immediate survival. Discounting naturally models this preference.
3. Handling Uncertainty: The future is inherently uncertain. The agent's model of the environment might be imperfect, or the environment itself might change unexpectedly. Applying a discount factor implicitly accounts for this increasing uncertainty over longer time horizons; rewards far in the future are less certain and thus valued less.

Choosing the Value of Gamma ($\gamma$)

The choice of $\gamma$ significantly impacts the agent's behavior:

• If $\gamma = 0$: The agent becomes completely 'myopic'. It only cares about maximizing the immediate reward $R_{t+1}$. It ignores all future consequences of its actions. The return is simply $G_t = R_{t+1}$.
• If $\gamma$ is close to 1 (e.g., 0.99): The agent is 'far-sighted'. Future rewards are valued almost as highly as immediate rewards. The agent considers the long-term consequences of its actions more heavily.
• If $\gamma$ is between 0 and 1: The agent balances immediate and future rewards. Smaller values prioritize short-term gain, while larger values emphasize long-term accumulation.

The specific value of $\gamma$ is often treated as a hyperparameter of the problem setup, chosen based on the nature of the task. For example, in financial applications, $\gamma$ might relate to prevailing interest rates. In episodic tasks, $\gamma=1$ (no discounting) is sometimes used if the episodes are guaranteed to terminate, although using $\gamma < 1$ can often lead to faster learning.

The following chart illustrates how the discount factor $\gamma^k$ diminishes the value of rewards over future time steps $k$ for different values of $\gamma$:

The weight given to a reward received $k$ steps in the future decreases exponentially with $k$. A higher $\gamma$ leads to slower decay, valuing future rewards more.

In Reinforcement Learning, the ultimate objective is typically to find a policy $\pi$ that maximizes the expected discounted return from each state. The concept of discounting is therefore fundamental to defining the very goal we are trying to achieve and enables the formalisms, like the value functions we'll discuss next, that allow us to solve MDPs.