Prediction Error as Curiosity: Intrinsic Motivation

Equipping an agent with intrinsic motivation provides a powerful exploration strategy in reinforcement learning, diverging from methods like UCB and Thompson Sampling which primarily focus on reasoning about value uncertainty. Instead of relying solely on external rewards provided by the environment ( $r^e$ ), which can be sparse or delayed, intrinsically motivated agents generate their own internal reward signals ( $r^i$ ) to guide exploration. A common form of intrinsic motivation is curiosity, where the agent is rewarded for encountering situations it finds surprising or unpredictable.

One prominent way to formalize curiosity is through prediction error. The core idea is simple: if the agent can accurately predict the consequences of its actions in a particular part of the state space, it likely understands that region well and has less incentive to explore it further. Conversely, if its predictions are poor, it indicates a gap in its knowledge, making that region "interesting" and worthy of exploration. The agent's intrinsic reward becomes proportional to its inability to predict the future.

The Intrinsic Curiosity Module (ICM)

A well-known implementation of this idea is the Intrinsic Curiosity Module (ICM). Instead of working directly with raw, high-dimensional states (like images), which might contain distracting information irrelevant to the agent's task, ICM operates in a learned feature space. This helps the agent focus its curiosity on aspects of the environment that it can actually influence or that affect its future trajectory.

The ICM typically consists of three main components, often implemented as neural networks:

Feature Encoder ( $\phi$ ): This network maps the raw state $s_t$ into a lower-dimensional feature vector $\phi(s_t)$ . The goal is to capture compact, relevant information about the state.
Forward Dynamics Model: Takes the current feature representation $\phi(s_t)$ and the agent's action $a_t$ , and predicts the feature representation of the next state, $\hat{\phi}(s_{t+1})$ . $\hat{\phi}(s_{t+1}) = f_{forward}(\phi(s_t), a_t)$
Inverse Dynamics Model: Takes the feature representations of the current state $\phi(s_t)$ and the next state $\phi(s_{t+1})$ , and predicts the action $\hat{a}_t$ that the agent took to transition between them. $\hat{a}_t = f_{inverse}(\phi(s_t), \phi(s_{t+1}))$

Here's how these components interact:

The Intrinsic Curiosity Module (ICM) architecture. Raw states are encoded into features. The forward model predicts the next feature based on the current feature and action. The inverse model predicts the action based on consecutive features. The prediction error of the forward model generates the intrinsic reward signal. Both the forward and inverse models are trained concurrently, with the inverse model loss also contributing to training the feature encoder.

Training the ICM:

The Forward Model is trained to minimize the difference between its predicted next feature representation $\hat{\phi}(s_{t+1})$ and the actual next feature representation $\phi(s_{t+1})$ computed by passing $s_{t+1}$ through the encoder. The loss function is typically the squared Euclidean distance: $L_{Forward} = \frac{1}{2} || \hat{\phi}(s_{t+1}) - \phi(s_{t+1}) ||^2_2$
The Inverse Model is trained to minimize the error in predicting the actual action $a_t$ taken. If actions are discrete, this is usually a cross-entropy loss; for continuous actions, it might be a mean squared error. $L_{Inverse} = \text{Loss}(\hat{a}_t, a_t)$
Crucially, the Feature Encoder ( $\phi$ ) is trained only by the gradients flowing back from the inverse dynamics loss $L_{Inverse}$ . It is not trained by the forward dynamics loss. Why? Training the encoder with the inverse model encourages it to learn features that are relevant to predicting the agent's own actions. Features that change unpredictably but cannot be influenced by the agent (like leaves rustling randomly on a tree in the background) are less likely to be encoded, preventing the agent from getting stuck trying to predict inherently unpredictable aspects of the environment (the "noisy TV problem").

Generating the Intrinsic Reward:

The intrinsic reward $r^i_t$ given to the RL agent at time $t$ is calculated based on the prediction error of the forward dynamics model in the feature space:

r^i_t = \frac{\eta}{2} || \hat{\phi}(s_{t+1}) - \phi(s_{t+1}) ||^2_2

Here, $\eta > 0$ is a scaling factor that controls the magnitude of the curiosity reward. A higher prediction error indicates a greater "surprise" for the agent, leading to a larger intrinsic reward, encouraging it to explore transitions whose outcomes it understands poorly.

Integrating with the RL Agent:

The intrinsic reward $r^i_t$ is typically added to the extrinsic reward $r^e_t$ received from the environment. The agent's policy (e.g., in A2C or PPO) is then trained to maximize the sum of discounted future rewards, where the reward at each step is $r_t = r^e_t + r^i_t$ . Sometimes, a weighting factor $\beta$ is used: $r_t = \beta r^e_t + (1-\beta) r^i_t$ . This combined reward signal motivates the agent to both achieve the external task objectives and satisfy its curiosity by exploring unfamiliar state-action transitions.

Advantages

Using prediction error as intrinsic motivation offers several benefits:

Dense Rewards: It provides a dense learning signal even when external rewards are sparse or non-existent, facilitating exploration in challenging environments.
Directed Exploration: It guides exploration towards areas where the agent's model is inaccurate, potentially leading to more efficient discovery of rewarding states compared to random exploration.
Focus on Controllable Dynamics: By using the inverse dynamics model to train the feature representation, it tends to focus exploration on aspects of the environment the agent can actually influence.

However, there are also points to consider:

Model Complexity: Implementing and training the ICM components adds complexity compared to standard RL algorithms.
Hyperparameter Sensitivity: The performance can be sensitive to the choice of network architectures, learning rates, the feature space dimensionality, and the scaling factor $\eta$ .
Potential for Distraction: While the inverse model helps, the agent might still be attracted to parts of the environment that are complex to predict but ultimately irrelevant to the main task if not designed carefully.

Prediction-error-based curiosity, exemplified by ICM, represents a significant step towards building agents that can actively explore and learn about complex environments in the absence of frequent external feedback. It's a powerful tool for tackling exploration challenges in scenarios ranging from robotic manipulation to navigating complex game worlds.

Was this section helpful?

References

Curiosity-driven Exploration by Self-supervised Prediction, Deepak Pathak, Pulkit Agrawal, Alexei A. Efros, Trevor Darrell, 2017 International Conference on Machine Learning (ICML) DOI: 10.48550/arXiv.1705.05363 - Introduces the Intrinsic Curiosity Module (ICM), a model that uses prediction error in a learned feature space to generate intrinsic rewards for exploration in deep reinforcement learning.
Reinforcement Learning: An Introduction, Richard S. Sutton, Andrew G. Barto, 2018 (The MIT Press) - A fundamental textbook that provides theoretical foundations of reinforcement learning, including principles of exploration and intrinsic motivation.
Exploration by Random Network Distillation, Yuri Burda, Harrison Edwards, Amos Storkey, Oleg Klimov, 2018 International Conference on Learning Representations (ICLR) DOI: 10.48550/arXiv.1810.12894 - Presents Random Network Distillation (RND), an alternative intrinsic motivation method that robustly addresses common challenges in curiosity-driven exploration, such as the noisy TV problem.