Building upon the core concepts of agents, environments, and their interactions, we now require a formal framework to represent the problems Reinforcement Learning addresses. This chapter introduces Markov Decision Processes (MDPs), the standard mathematical tool for modeling sequential decision-making where outcomes are partly random and partly under the control of a decision-maker.

You will learn to define the key elements of an MDP:

• The set of possible states ($S$) the environment can be in.
• The set of actions ($A$) the agent can take.
• The state transition probabilities ($P$), defining the dynamics of the environment ($P(s'|s, a)$).
• The reward function ($R$), specifying the immediate feedback ($R(s, a, s')$).
• The discount factor ($\gamma$), managing the importance of future rewards.

We will explore how an agent's behavior is defined by a policy ($\pi$) and how to evaluate the "goodness" of states and state-action pairs using value functions ($V^\pi$ and $Q^\pi$). This leads to understanding the goal of RL within the MDP framework: finding an optimal policy ($\pi^*$) that maximizes expected cumulative rewards.