So far, we have explored methods like Monte Carlo and Temporal Difference learning that rely on maintaining a table mapping each state or state-action pair to a value. This approach works well for problems with a manageable number of states. However, many interesting problems involve state spaces that are enormous or even continuous, making tabular methods impractical due to memory and computational requirements.

This chapter addresses this challenge by introducing function approximation. Instead of storing exact values for every state, we will approximate the value function using a parameterized function with far fewer parameters than the number of states. We aim to generalize from seen states to unseen ones.

You will learn:

• Why function approximation is necessary for large-scale Reinforcement Learning problems.
• How to represent states using feature vectors suitable for function approximators.
• Methods for approximating value functions, focusing initially on linear functions.
• How to use gradient descent, specifically semi-gradient methods, to update the parameters of your approximator based on RL experience.
• An introduction to using non-linear function approximators, such as neural networks, to capture more complex value function shapes.

By the end of this chapter, you will understand how to apply RL algorithms to problems where the state space is too large to handle with simple tables.