In the previous chapter, we formalized sequential decision-making problems using Markov Decision Processes (MDPs). We defined states, actions, rewards, and the goal of maximizing cumulative future rewards (return). A central idea was the concept of value functions, which quantify how good it is for an agent to be in a particular state ($V^{\pi}(s)$) or to take a specific action in a state ($Q^{\pi}(s,a)$) while following a policy $\pi$.

This chapter focuses on methods for calculating these value functions. We will introduce the Bellman equations, which express the value of a state or state-action pair in terms of the expected values of successor states. These equations form the basis for many RL algorithms.

Specifically, you will learn about:

• The Bellman expectation equation, which relates the value function of a policy $\pi$ to itself recursively.
• The Bellman optimality equation, which defines the value function for the optimal policy $\pi^*$.
• Dynamic Programming (DP) methods, specifically Policy Iteration and Value Iteration, which can compute optimal policies and value functions given a complete model of the environment (the MDP).
• The limitations of DP, particularly its reliance on a known model, motivating the model-free methods discussed in later chapters.

By the end of this chapter, you will understand how to theoretically compute optimal value functions and policies when the environment's dynamics are fully known.