Practice : Adversarial Testing

Purpose and Strategic Importance

Standard evaluation suites test whether a model performs well on typical inputs drawn from the same distribution as its training data. They do not test whether the model is robust to unusual, manipulated, or malicious inputs — the conditions that matter most for safety in production. Adversarial testing deliberately challenges AI systems with inputs designed to expose their vulnerabilities: edge cases at the boundary of learned behaviour, inputs that confuse or mislead the model, and deliberate attempts to manipulate model outputs through carefully crafted perturbations.

This practice is essential for any AI system deployed in a real-world context where inputs are not controlled. Users will encounter the model with inputs that differ from the training distribution; malicious actors will actively attempt to manipulate model outputs. Adversarial testing surfaces the failure modes that would otherwise only be discovered — at significant cost — in production.

Description of the Practice

  • Designs adversarial test cases that cover edge cases, distribution shifts, input perturbations, and deliberate manipulation attempts specific to the model's domain and deployment context.
  • Applies established adversarial attack methods — FGSM, PGD, or domain-specific approaches — to generate inputs that are designed to cause misclassification or unexpected outputs.
  • Tests for model sensitivity to semantically irrelevant input variations — typographical noise, formatting changes, paraphrase in language models — that should not change model outputs but often do.
  • Includes adversarial test results in model evaluation reports and uses them to inform deployment decisions and mitigation strategies.
  • Feeds findings from adversarial testing back into the training process through adversarial training or data augmentation to improve model robustness.

How to Practise It (Playbook)

1. Getting Started

  • Identify the adversarial scenarios most relevant to your model's use case: for a text classifier, paraphrase attacks and negation handling; for a vision model, perturbation and occlusion; for a recommendation system, manipulation of input features.
  • Build a seed adversarial test suite for your most important production model, covering the scenarios identified and establishing a baseline robustness profile.
  • Integrate adversarial testing into the pre-deployment evaluation pipeline alongside standard evaluation, making it a standard component of the model release process.
  • Review adversarial test failures carefully to distinguish genuine robustness issues from test artefacts, and prioritise mitigation for failures with plausible real-world analogues.

2. Scaling and Maturing

  • Automate adversarial test generation for common attack types, reducing the manual effort required to maintain comprehensive adversarial coverage as models evolve.
  • Build domain-specific adversarial scenarios based on production incident data — when real users find ways to trigger unexpected model behaviour, add those patterns to the adversarial test suite.
  • Implement adversarial training for models where robustness is a critical requirement, incorporating adversarial examples into the training distribution to improve generalisation to edge cases.
  • Develop a robustness benchmark specific to your domain that tracks adversarial robustness over model versions, enabling detection of robustness regressions alongside accuracy tracking.

3. Team Behaviours to Encourage

  • Approach adversarial testing with the mindset of an attacker, not a developer — the goal is to break the model, not to confirm that it works for expected inputs.
  • Involve domain specialists in adversarial scenario design, as they have the best understanding of the edge cases and manipulation attempts that real users or malicious actors might attempt.
  • Report adversarial test results honestly, including significant failure modes, even when they are difficult to mitigate before a planned deployment date.
  • Use adversarial testing findings to inform user-facing communication about model limitations, ensuring that users understand where the model should not be trusted.

4. Watch Out For…

  • Adversarial testing that is too narrow — covering only the most well-known attack types — while missing domain-specific vulnerabilities that are more likely to manifest in production.
  • Treating a model as "robust" because it passes a standard adversarial benchmark without considering whether the benchmark is relevant to your specific deployment context.
  • Adversarial testing conducted as a pre-deployment exercise but never repeated after retraining, missing new vulnerabilities introduced by changes in training data or architecture.
  • Findings from adversarial testing that are documented but never actioned, creating a record of known vulnerabilities without the investment to address them.

5. Signals of Success

  • Adversarial testing is a standard component of every model evaluation pipeline, with results reviewed and considered in deployment decisions.
  • The team maintains a growing adversarial test suite informed by production incidents and ongoing security research, not a static set of initial test cases.
  • Adversarial testing has identified at least one significant vulnerability that was subsequently mitigated before deployment, demonstrating the practice's protective value.
  • Model robustness metrics are tracked over model versions, with regressions triggering investigation and mitigation before deployment.
  • Security and domain experts participate in adversarial scenario design, bringing perspectives beyond the data science team to bear on vulnerability identification.