Reward Functions

The reward function defines the goal in a Markov Decision Process. It is an essential element that interacts with states, actions, and the environment's dynamics (transition probabilities).

Think of the reward function as the way we communicate the objective to the learning agent. In each step of the interaction, after the agent performs an action $a_t$ in state $s_t$ and the environment transitions to a new state $s_{t+1}$, the environment provides a numerical reward signal, $r_{t+1}$. This signal indicates how good or bad that specific transition was from the perspective of the task objective.

Formally, the reward function $R$ specifies this immediate feedback. It can take different forms depending on what information the reward depends on:

1. State, Action, and Next State: The most general form is $R(s, a, s')$, where the reward depends on the starting state $s$, the action $a$ taken, and the resulting state $s'$. The expected reward for taking action $a$ in state $s$ can be written as: $r(s, a) = \mathbb{E}[R_{t+1} | S_t = s, A_t = a] = \sum_{s'} P(s'|s, a) R(s, a, s')$ Here, $R_{t+1}$ is the random variable representing the reward at time $t+1$, and $P(s'|s, a)$ is the state transition probability we discussed earlier.

2. State and Action: Often, the reward is simplified and only depends on the state $s$ and the action $a$ taken, denoted as $R(s, a)$.

3. State Only: In some scenarios, particularly goal-based tasks, the reward might only depend on reaching certain states, denoted $R(s')$. For example, a large positive reward for reaching a target state and zero otherwise.

The reward function is central to RL because it implicitly defines what the agent should strive to achieve. The agent's objective isn't necessarily to maximize the immediate reward $r_{t+1}$, but rather the cumulative reward over time, which we previously defined as the return ($G_t$). The discount factor $\gamma$ plays a significant role here, balancing the importance of immediate rewards versus future rewards when calculating the return.

The environment provides a reward $r_{t+1}$ along with the next state $s_{t+1}$ after the agent takes action $a_t$ in state $s_t$. This reward signal is the primary feedback used by the agent to learn a desirable policy.

Designing Reward Functions

Specifying the reward function is a form of reward engineering. It's how we translate a high-level goal into a concrete signal the agent can use for learning. This isn't always straightforward:

• Sparse Rewards: In many problems, rewards might be zero for most steps, with a non-zero reward only occurring upon reaching a terminal state (e.g., winning or losing a game). Learning can be slow with sparse rewards as feedback is infrequent.
  • Example: Chess: +1 for checkmate, -1 for being checkmated, 0 for all other moves.
• Dense Rewards: Providing smaller, more frequent rewards can guide the agent more effectively. However, poorly designed dense rewards can lead to unintended agent behavior, where the agent optimizes for intermediate rewards rather than the true objective.
  • Example: Robot navigating a maze: +10 for reaching the goal, -0.1 for each step taken (encourages finding shorter paths), -5 for hitting a wall.

The Reward Hypothesis is a fundamental concept in RL: it posits that all goals and purposes can be characterized as the maximization of the expected value of the cumulative sum of a received scalar signal (the reward). Getting the reward function right is therefore essential for successful reinforcement learning. It must accurately reflect the actual goal of the task. If the reward function incentivizes behavior different from what you truly want, the agent will likely learn that unintended behavior.

Understanding the reward function is fundamental before moving on to how agents actually use these rewards (along with state transitions) to evaluate policies and find optimal ways to behave. The interaction between the reward signal, the environment's dynamics, and the agent's policy is the core of the learning process in MDPs.