In the previous chapter, we examined Monte Carlo methods, which update value estimates only after an entire episode completes. This requires waiting until the final outcome is known before any learning occurs for that episode. Temporal-Difference (TD) learning offers a different approach, enabling updates after each step.

TD methods learn directly from experience, similar to MC methods. However, unlike MC, they update value estimates based partially on other currently learned estimates, without waiting for the episode's final outcome. This technique is known as bootstrapping. It allows TD methods to learn from incomplete episodes and often converges faster than MC methods in practice.

This chapter introduces the fundamentals of TD learning. We will cover:

• The basic TD prediction algorithm, TD(0), for estimating the state-value function $V_\pi$.
• The advantages of TD learning compared to Monte Carlo methods.
• Two primary TD control algorithms: SARSA (State-Action-Reward-State-Action), an on-policy method, and Q-Learning, a popular off-policy method. We will analyze their update rules and key differences.
• An introduction to Expected SARSA.
• Practical implementation of the Q-learning algorithm to solve a simple control task.

By the end of this chapter, you will understand how TD methods operate and be able to implement core TD algorithms for prediction and control problems.