Attacks on Reinforcement Learning Agents

Reinforcement Learning (RL) presents a distinct set of challenges and opportunities for adversarial attacks compared to supervised learning paradigms like computer vision or natural language processing. Unlike models that process static inputs, RL agents operate within an interactive loop, making sequential decisions based on observations from an environment to maximize a cumulative reward signal. This dynamic interaction opens up unique attack vectors.

Adversarial attacks against RL agents aim to manipulate the agent's learned policy, $\pi(a|s)$, causing it to take suboptimal or malicious actions. These attacks can be broadly categorized based on when they occur (during training or testing) and what component of the RL loop they target (observations, rewards, or actions).

Test-Time Attacks on RL Agents

Test-time attacks, analogous to evasion attacks in supervised learning, occur after the agent has been trained and deployed. The attacker's goal is typically to perturb the agent's observations (states) minimally such that the agent selects significantly worse actions than it would normally.

Consider an agent interacting with an environment. At each timestep $t$, the agent observes state $s_t$, selects an action $a_t \sim \pi(a|s_t)$, receives a reward $r_t$, and transitions to the next state $s_{t+1}$. An attacker might introduce a small perturbation $\delta_t$ to the observed state $s_t$, resulting in an adversarial state $s'_t = s_t + \delta_t$. The agent then acts based on this perturbed state: $a'_t \sim \pi(a|s'_t)$.

The attacker's objective can be formulated in several ways, often aiming to minimize the expected cumulative reward (or maximize a cost function) starting from the perturbed state. For an attack at timestep $t$, the objective might be to find a perturbation $\delta_t$ with bounded norm (e.g., $||\delta_t||_p \leq \epsilon$) that minimizes the expected return:

$$\min_{\delta_t: ||\delta_t||_p \leq \epsilon} \mathbb{E} \left[ \sum_{k=t}^{\infty} \gamma^{k-t} r_k \mid s'_t = s_t + \delta_t, a_k \sim \pi(a|s'_k) \text{ for } k \ge t \right]$$

Where $\gamma$ is the discount factor.

If the policy $\pi$ is differentiable with respect to its input state (common in deep RL using neural networks), gradient-based methods like FGSM or PGD can be adapted. For example, using the value function $V(s)$ or Q-function $Q(s, a)$ which estimate the expected return, an attacker could compute the gradient with respect to the state input and perturb the state in the direction that decreases the expected value.

Example: FGSM-like Attack on Value Function

If the agent uses a value function $V_\theta(s)$ parameterized by $\theta$, an attacker could craft a perturbation $\delta$ to minimize the predicted value:

$$\delta = -\epsilon \cdot \mathrm{sign}(\nabla_s V_\theta(s))$$ $$s' = s + \delta$$

This aims to make the agent perceive the current state as less valuable, potentially leading to suboptimal actions. Similar approaches can target the Q-function or the policy network directly.

Challenges in test-time RL attacks include:

• Temporal Consistency: Perturbations might need to be applied consistently over multiple timesteps to achieve a lasting effect.
• Partial Observability: In Partially Observable Markov Decision Processes (POMDPs), the agent acts based on observations $o_t$, which may not fully reveal the true state $s_t$. Attacks must perturb observations effectively despite this incomplete information.
• Non-Differentiable Policies/Environments: If parts of the policy or environment simulation are not differentiable, gradient-based methods fail. Attackers might resort to score-based or decision-based black-box techniques, querying the agent and observing outcomes to estimate gradients or find vulnerabilities.

Training-Time Attacks on RL Agents

Training-time attacks, similar to data poisoning, target the learning process itself. The attacker manipulates the agent's training experience to embed vulnerabilities or degrade performance.

Attack Vectors:

1. Observation Poisoning: The attacker subtly modifies the states or observations presented to the agent during training. This can steer the agent towards learning incorrect state-action mappings or embedding backdoors.
2. Reward Poisoning: The attacker manipulates the reward signal $r_t$ received by the agent. By providing misleading rewards, the attacker can incentivize the agent to learn a policy that serves the attacker's goals (e.g., reaching a specific unsafe state) or simply performs poorly overall.
3. Action Poisoning: In some settings (e.g., imitation learning or offline RL), the attacker might control or modify the actions taken during data collection, leading the agent to learn from flawed trajectories.

Example: Backdoor via Reward Poisoning

An attacker could modify the reward function during training such that the agent receives unusually high rewards for reaching a specific, otherwise undesirable, state only when a subtle trigger is present in the observation. For instance, in a self-driving car simulation, a specific rare visual pattern (the trigger) in the observation, paired with reaching a dangerous location, could be artificially rewarded during training. After deployment, if the agent observes this trigger, its poisoned policy might drive it towards the dangerous location.

The diagram below illustrates points where an attacker might intervene in the standard RL loop:

Potential attack points in the RL agent-environment interaction loop. Red 'X' indicates test-time perturbation of observations. Orange 'X' indicates training-time poisoning of observations or rewards.

Training-time attacks are often more powerful but harder to execute, as they typically require influence over the training environment or data generation process. Defending against these attacks involves learning algorithms, anomaly detection in training data (states, rewards), and secure environment design.

Understanding these RL-specific attack vectors is essential for developing secure autonomous systems. Defenses often involve policy optimization techniques, adversarial training adapted for sequential decisions, and methods to detect or filter poisoned experiences. The evaluation of robustness in RL also requires careful consideration of the sequential and interactive nature of the problem.