Certain defense methods, such as adversarial training and certified defenses, strive to enhance a machine learning model's fundamental strength. However, some other defense techniques inadvertently depend on a phenomenon called gradient masking or gradient obfuscation. Grasping this problem is important for accurately assessing model security and preventing a misleading perception of model strength.
Gradient masking occurs when a defense mechanism makes it difficult for gradient-based attacks, such as PGD, to find effective adversarial perturbations, even though the underlying vulnerability might still exist. Instead of truly making the model robust, the defense essentially "hides" or "obfuscates" the gradients that these attacks rely on.
How Gradient Masking Manifests
There are several ways defenses can cause gradient masking:
- Shattered Gradients: Some defenses introduce operations that lead to numerically unstable or zero gradients almost everywhere. For example, extremely high-frequency functions or networks relying heavily on saturated activation functions might exhibit this. An attacker trying to compute gradients gets values that are either zero,
NaN (Not a Number), or Inf (Infinity), rendering standard gradient descent ineffective.
- Stochastic Gradients: Defenses incorporating randomness, either in the model architecture (e.g., dropout at test time) or through random input transformations, can produce noisy gradient estimates. Averaging gradients over multiple runs might help, but the noise can still significantly hinder the optimization process for finding adversarial examples, especially for iterative attacks like PGD that rely on consistent gradient directions. Randomized smoothing, while stochastic, aims for certified robustness, which is a different mechanism than simply using randomness to break gradient computations.
- Non-differentiable Operations: Techniques that involve non-differentiable steps, such as certain forms of input quantization, thresholding, or complex pre-processing, explicitly break the gradient flow. While the model might still be vulnerable, gradient-based attacks cannot directly optimize perturbations through these non-differentiable layers.
Why Obfuscated Gradients are Problematic
The primary danger of gradient masking is the false sense of security it provides. A model defended using a technique that relies heavily on gradient obfuscation might show high accuracy against standard gradient-based attacks (like FGSM or PGD) in initial evaluations. This suggests the defense is effective.
However, this robustness is often superficial. The model hasn't learned to classify inputs correctly near the decision boundary; it has simply made it harder for specific attack algorithms to exploit the weaknesses using gradients. More sophisticated or different types of attacks can often bypass these defenses:
- Different Optimization Techniques: Optimization-based attacks like Carlini & Wagner (C&W) might be less sensitive to small gradient inaccuracies.
- Transfer Attacks: Adversarial examples crafted against an undefended (or differently defended) substitute model might still successfully transfer to the model with obfuscated gradients.
- Score-Based and Decision-Based Attacks: Attacks that do not rely on gradients, such as Boundary Attack or techniques using only model output scores, can potentially succeed where gradient-based methods fail.
- Adaptive Attacks: An attacker aware of the defense mechanism can specifically design attacks to circumvent the gradient obfuscation. For instance, if a defense uses non-differentiable components, an attacker might use techniques like the Backward Pass Differentiable Approximation (BPDA) to estimate gradients or simply target earlier layers before the non-differentiable step.
The following diagram provides an illustration of how gradient masking might affect the loss explored by an attacker.
A view comparing a smooth loss area where gradients guide attacks effectively, versus an obfuscated area where gradients are unreliable, potentially misleading evaluations based solely on gradient attacks.
Detecting and Overcoming Gradient Masking
Detecting gradient masking requires rigorous evaluation that goes past standard PGD tests. As detailed in the next chapter on evaluation:
- Test with Diverse Attacks: Include optimization-based (C&W), score-based, and decision-based attacks in your evaluation suite.
- Analyze Transferability: Check if attacks generated on other models transfer easily. High transferability might indicate the defense isn't fundamentally strong.
- Implement Adaptive Attacks: Specifically design attacks that account for the defense mechanism. If the defense involves randomization, average gradients or use expectation-over-transformation attacks. If it involves non-differentiable parts, try to approximate gradients or attack preceding layers.
- Check White-Box vs. Black-Box Performance: If a defense performs exceptionally well against white-box gradient attacks but poorly against black-box attacks (especially transfer or query-based), it's a strong indicator of obfuscated gradients.
In summary, while gradient masking might appear to enhance robustness against naive attacks, it often represents a failure mode rather than a successful defense strategy. True adversarial robustness requires models to fundamentally resist perturbations, not just to make the optimization difficult for specific attackers to navigate using gradients. Proper evaluation using a diverse set of strong, adaptive attacks is essential to distinguish genuine robustness from mere gradient obfuscation.
