The Reinforcement Learning Problem Setup

Okay, let's begin by revisiting the basic structure of a Reinforcement Learning problem. As mentioned in the chapter introduction, RL is fundamentally about learning through interaction. At its heart, the RL setup involves two primary components: the agent and the environment.

• The Agent is the learner or decision-maker. Think of it as the algorithm or controller you are trying to build. It observes the environment and decides which actions to take.
• The Environment is everything outside the agent that it interacts with. It represents the problem or task the agent needs to solve. The environment responds to the agent's actions by changing its state and providing feedback.

This interaction unfolds over a sequence of discrete time steps, $t = 0, 1, 2, ...$. At each time step $t$, the agent observes the current situation of the environment, which is represented as a state, denoted by $S_t$. A state is a description of the environment that provides the necessary information for the agent to make a decision. This could be anything from the position of pieces on a chessboard to sensor readings from a robot or pixel data from a game screen.

Based on the observed state $S_t$, the agent selects an action, $A_t$, from a set of available actions. An action is a choice the agent can make to influence the environment. Examples include moving a robot arm, making a move in a game, or adjusting a thermostat.

After the agent performs action $A_t$ in state $S_t$, the environment transitions to a new state, $S_{t+1}$, at the next time step. Along with the new state, the environment provides a numerical reward, $R_{t+1}$, to the agent. This reward signal indicates how good or bad the action $A_t$ taken in state $S_t$ was in the short term. Positive rewards typically correspond to desirable outcomes, while negative rewards (or costs) signify undesirable ones.

The agent's behavior is defined by its policy, denoted by $\pi$. The policy is the agent's strategy, mapping states to actions. It dictates which action the agent chooses when it finds itself in a particular state. A policy can be deterministic, written as $a = \pi(s)$, meaning it always outputs the same action for a given state. Or it can be stochastic, written as $\pi(a|s) = P(A_t = a | S_t = s)$, specifying the probability of taking each possible action $a$ in state $s$.

The overarching goal of the agent is not just to maximize the immediate reward $R_{t+1}$, but to maximize the total amount of reward it accumulates over the long run. This cumulative reward, starting from time step $t$, is often called the return, denoted $G_t$. A common formulation is the discounted return:

$$G_t = R_{t+1} + \gamma R_{t+2} + \gamma^2 R_{t+3} + \dots = \sum_{k=0}^{\infty} \gamma^k R_{t+k+1}$$

Here, $\gamma$ is the discount factor, a number between 0 and 1 ($0 \le \gamma \le 1$). It determines the present value of future rewards. A $\gamma$ close to 0 makes the agent "myopic," focusing mainly on immediate rewards, while a $\gamma$ close to 1 makes the agent "farsighted," striving for high total reward over the long term. The discount factor also helps keep the return finite in tasks that might continue indefinitely.

This continuous cycle of state-action-reward forms the core interaction loop in Reinforcement Learning.

The agent-environment interaction loop. The agent observes the state, selects an action, the environment transitions to a new state and provides a reward, which the agent then observes to inform future actions.

Understanding these fundamental components agent, environment, state, action, reward, and policy is essential. The algorithms we'll discuss in subsequent chapters, like Q-learning, DQN, REINFORCE, and Actor-Critic methods, all operate within this framework, providing different ways for the agent to learn an effective policy that maximizes its expected return. The limitations of representing policies or value functions directly in tables for large problems, which we'll touch upon next, motivate the need for the function approximation techniques central to this course.