Action and Observation Space Representation

How an agent perceives its environment (observation space) and the set of actions it can perform (action space) are fundamental aspects of any reinforcement learning problem. The way we structure and process these spaces within our implementation directly influences the agent's ability to learn effectively and efficiently. Getting the representation right is a significant step towards building successful RL systems.

Representing Observation Spaces

The observation space defines what information the agent receives at each step. The nature of this information dictates the appropriate network architecture and preprocessing techniques.

Vector Observations

These are often the simplest form, represented as a flat vector of numerical values. Examples include sensor readings, joint angles in robotics, or curated features from a game state.

Representation: Typically fed directly into fully connected layers (Multi-Layer Perceptrons, MLPs).
Preprocessing: Normalization or standardization is highly recommended. Features can have very different scales (e.g., position in meters vs. velocity in meters/second). Scaling inputs to have zero mean and unit variance, or scaling them to a fixed range like $[-1, 1]$ or $[0, 1]$ , helps stabilize training and prevents features with larger magnitudes from dominating the learning process. Running statistics (mean and standard deviation) can be maintained and used for normalization, updating them as more data comes in or pre-calculating them from a sample dataset if available.

Image Observations

Common in game environments (like Atari) or robotics tasks involving cameras, observations are represented as pixel matrices.

Representation: Convolutional Neural Networks (CNNs) are the standard choice for extracting spatial hierarchies of features from images. The output of the CNN's convolutional layers is usually flattened and then passed to fully connected layers for policy or value estimation.
Preprocessing: Several standard techniques improve performance and efficiency:
- Resizing: Reduce image dimensions (e.g., to 84x84) to decrease computational load.
- Grayscaling: Convert RGB images to grayscale if color information is not necessary, reducing input channels from 3 to 1.
- Frame Stacking: Stack several consecutive frames (e.g., 4 frames) together as a single observation. This allows the agent, even with a feedforward network like a CNN, to infer temporal information like velocity and acceleration, which is often critical for making informed decisions.
- Normalization: Pixel values (typically 0-255) should be normalized, often by dividing by 255.0 to scale them to the $[0, 1]$ range.

Sequential Observations

When observations have temporal dependencies not captured by simple frame stacking, like in natural language processing tasks or time-series analysis, specialized architectures are needed.

Representation: Recurrent Neural Networks (RNNs), Long Short-Term Memory networks (LSTMs), or Gated Recurrent Units (GRUs) are commonly used to process sequential data and maintain an internal state. More recently, Transformer architectures have also shown promise in RL tasks with long-range dependencies.

Multi-Modal Observations

Some tasks provide observations from multiple sources (e.g., camera images plus robot joint angles, or game state vectors plus textual information).

Representation: Requires designing network architectures that can process each modality appropriately (e.g., CNN for images, MLP for vectors) and then combine the resulting feature representations (e.g., by concatenation or more sophisticated attention mechanisms) before feeding them into the final decision-making layers.

Representing Action Spaces

The action space defines the set of possible actions the agent can take. The representation strategy depends heavily on whether the actions are discrete, continuous, or a mix.

Discrete Action Spaces

The agent chooses one action from a finite set (e.g., up, down, left, right, button_A).

Representation:
- Value-based methods (DQN): The network outputs a Q-value for each possible action. The action selection (e.g., epsilon-greedy) typically involves choosing the action with the highest Q-value. The output layer has $N$ neurons, where $N$ is the number of discrete actions.
- Policy gradient/Actor-Critic methods (PPO, A2C): The network (actor) outputs probabilities (or logits, which are unnormalized log-probabilities) for each action. The output layer has $N$ neurons, typically followed by a softmax activation to produce probabilities for sampling. Action selection involves sampling from this probability distribution.

Continuous Action Spaces

The agent chooses actions represented by real-valued numbers within certain bounds (e.g., steering angle $[-1, 1]$ , motor torque $[-5, 5]$ ). Actions often form a vector (e.g., torques for multiple joints).

Representation:
- Stochastic Policies (PPO, SAC, A2C): The actor network typically outputs parameters of a probability distribution. For a Gaussian distribution (common choice), the network outputs the mean ( $\mu$ ) and standard deviation ( $\sigma$ ) for each action dimension. The actual action is then sampled from this distribution, $a \sim \mathcal{N}(\mu(s), \sigma(s))$ . The standard deviation can be state-dependent (output by the network) or state-independent (learned as separate parameters or fixed).
- Deterministic Policies (DDPG): The actor network directly outputs the action value for each dimension. Exploration is usually added externally (e.g., by adding noise like Ornstein-Uhlenbeck or Gaussian noise to the output action during training).
Action Bounding: Continuous actions often have defined limits. A common technique is to have the network output unbounded values and then pass them through a squashing function, like the hyperbolic tangent function ( $tanh$ ), which maps values to the range $[-1, 1]$ . The result can then be scaled and shifted to match the environment's specific action bounds. For example, if the environment requires actions in $[a_{min}, a_{max}]$ , the output $a_{net}$ from the $tanh$ can be transformed: $a_{env} = a_{min} + \frac{(a_{net} + 1)(a_{max} - a_{min})}{2}$ Using $tanh$ is particularly important in algorithms like Soft Actor-Critic (SAC), where the mathematics accounts for the change in probability density caused by the squashing function.
Normalization: Similar to observations, normalizing continuous action targets during critic training in actor-critic methods can sometimes be beneficial, although applying normalization directly to policy outputs requires careful consideration.

Hybrid and Multi-Discrete Action Spaces

More complex scenarios involve combinations:

Multi-Discrete: The agent needs to select one option from several independent discrete sets simultaneously (e.g., choose a weapon AND choose a movement direction). Often handled by having separate output heads for each discrete choice.
Hybrid: The agent chooses a discrete action type, and based on that type, selects continuous parameters (e.g., choose 'move' action, then specify continuous velocity and direction). These require more complex network architectures and action parameterization, sometimes involving conditional outputs or hierarchical policies.

Network output structures depend on the action space type. Discrete spaces typically use a single head outputting probabilities or Q-values, while continuous spaces might output distribution parameters or direct action values.

General Notes

Environment Libraries: Libraries like Gymnasium (formerly OpenAI Gym) provide standardized ways to define observation_space and action_space objects. These objects contain information about the shape, data type (e.g., float32, uint8), and bounds (min/max values) of the spaces. Leveraging these standards simplifies implementation and promotes code reusability.
Consistency: Ensure that the preprocessing applied during training (e.g., normalization, frame stacking) is also applied consistently during evaluation or deployment.
Normalization Layers: Consider using network layers that perform normalization (like LayerNorm or BatchNorm) within the architecture itself, which can sometimes adapt better than fixed normalization based on running statistics.

Carefully choosing how to represent and preprocess observations and actions is not just a low-level implementation detail; it's an integral part of designing an RL agent that can effectively learn from its interactions. Experimentation is often necessary to find the representation that works best for a specific task and algorithm.

Was this section helpful?

References

Reinforcement Learning: An Introduction, Richard S. Sutton and Andrew G. Barto, 2018 (MIT Press) - Provides fundamental definitions of states, observations, and actions, and a comprehensive overview of classic and modern RL algorithms, including discussions on function approximation for large spaces.
Human-level control through deep reinforcement learning, Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Alex Graves, Ioannis Antonoglou, Daan Wierstra, and Martin Riedmiller, 2015 Nature, Vol. 518 DOI: 10.1038/nature14236 - Introduces Deep Q-Networks (DQN), demonstrating the use of CNNs for image observations, preprocessing techniques like resizing, grayscaling, and frame stacking, and addresses discrete action spaces.
Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor, Tuomas Haarnoja, Aurick Zhou, Kristian Hartikainen, George Tucker, Sehoon Ha, Jie Tan, Vikash Kumar, Henry Zhu, Abhishek Gupta, Pieter Abbeel, and Sergey Levine, 2018 International Conference on Machine Learning (ICML) DOI: 10.48550/arXiv.1801.01290 - A prominent algorithm for continuous action spaces, discussing stochastic policies and explicitly detailing the use of the tanh squashing function for action bounding, a primary implementation detail mentioned.
Decision Transformer: Reinforcement Learning via Sequence Modeling, Lili Chen, Kevin Lu, Aravind Rajeswaran, Kimin Lee, Aditya Grover, Michael Laskin, Pieter Abbeel, Aravind Srinivas, Igor Mordatch, 2021 Advances in Neural Information Processing Systems (NeurIPS), Vol. 34 DOI: 10.48550/arXiv.2106.01345 - Introduces the Decision Transformer architecture, demonstrating how reinforcement learning tasks can be framed as sequence modeling problems, using Transformer networks for handling temporal dependencies.
Gymnasium Documentation, The Gymnasium Contributors, 2024 - Official documentation for standardized environment interfaces in reinforcement learning, detailing observation_space and action_space objects.