In the preceding chapters, we established the framework of Markov Decision Processes (MDPs) and used Dynamic Programming methods to find optimal policies. A key limitation of DP is the requirement of a complete model of the environment, including state transition probabilities and reward functions. Often, such a model is unavailable.

This chapter introduces Monte Carlo (MC) methods, a class of model-free Reinforcement Learning algorithms. MC methods learn directly from episodes of experience, without needing prior knowledge of the environment's dynamics. They operate by averaging the sample returns obtained from interaction sequences (episodes).

Here, you will focus on:

• The fundamental idea of learning from complete episodes.
• Using MC for prediction: estimating the state-value function $V^\pi$.
• Applying MC for control: finding optimal policies by estimating the action-value function $Q^\pi$.
• Understanding the difference between on-policy and off-policy MC methods.
• Addressing the need for exploration in MC control, for example, using $\epsilon$-soft policies.
• Implementing basic MC prediction.

By working through MC methods, you'll gain insight into learning optimal behavior purely from sampled experience, a necessary step towards tackling problems where the environment's rules are unknown.