Practice : AI Incident Response

Purpose and Strategic Importance

AI incidents are distinct from conventional software incidents in important ways. They can be difficult to detect — model performance degradation is often gradual and statistical rather than a binary failure. They can be difficult to diagnose — the cause of unexpected model behaviour may lie in training data, feature computation, model version, or upstream data drift. And they can have ethical dimensions — a model making systematically biased decisions or producing harmful outputs requires a response that goes beyond technical remediation to address potential harm to affected users.

A structured AI incident response practice ensures that when AI systems fail — and they will — the team responds effectively and proportionately. This means detecting failures quickly, containing harm, communicating transparently, resolving root causes, and learning systematically so that similar incidents are less likely to recur. Without this practice, incident response is ad hoc, slow, and unlikely to produce the organisational learning needed to improve AI system reliability over time.

Description of the Practice

  • Classifies AI incidents by severity and type — technical failures, performance degradation, fairness violations, safety incidents, and ethical incidents — with response procedures tailored to each category.
  • Maintains incident runbooks that guide on-call engineers through containment, diagnosis, and resolution steps for common AI incident types.
  • Defines communication protocols for AI incidents, including who is notified, when, and what information they are given — covering internal escalation, user communication, and regulatory notification where required.
  • Conducts blameless post-incident reviews (PIRs) after every significant AI incident, producing written records of root cause analysis and systematic improvement actions.
  • Tracks incident metrics — frequency, severity, time to detect, time to resolve — to measure response effectiveness and identify systemic reliability issues.

How to Practise It (Playbook)

1. Getting Started

  • Define AI incident severity levels appropriate to your organisation — typically covering technical impact (system availability, error rate) and harm impact (user impact, fairness, safety) dimensions.
  • Write runbooks for the most likely AI incident types in your system — model performance degradation, data pipeline failures, and unexpected output distribution shifts.
  • Establish on-call rotation and escalation paths for AI systems, ensuring that every production model has a named on-call owner and a clear escalation chain.
  • Define communication templates for notifying affected users and internal stakeholders, so that communication under pressure is fast, accurate, and consistent.

2. Scaling and Maturing

  • Build incident management tooling integration — connecting monitoring alerts directly to incident management systems — to reduce the time from detection to response initiation.
  • Develop a library of past incidents, runbooks, and PIRs that serves as institutional memory for AI incident response, helping the team handle novel incidents by analogy with past experience.
  • Introduce tabletop exercises and simulated incidents to test response procedures and build team confidence without the pressure of a real production incident.
  • Extend incident response to cover third-party AI systems and models used by the organisation, ensuring that dependency on external AI does not create unmanaged risk.

3. Team Behaviours to Encourage

  • Treat AI incidents as learning opportunities rather than failures, creating a culture where incidents are reported and investigated honestly rather than minimised or concealed.
  • Prioritise harm containment over diagnosis when responding to incidents — roll back, disable, or constrain the system first; understand why second.
  • Complete post-incident reviews for all significant incidents and follow through on the improvement actions identified, closing the learning loop rather than leaving actions as recommendations that are never implemented.
  • Include fairness and harm assessment as standard components of incident review, not as optional additions reserved for obviously ethical incidents.

4. Watch Out For…

  • Incident response processes designed for conventional software failures that miss the specific characteristics of AI incidents — gradual degradation, statistical manifestation, and ethical dimensions.
  • Blameful incident cultures that discourage honest reporting and investigation, preventing the organisational learning that makes response better over time.
  • Post-incident reviews that identify root causes but produce improvement actions that are never prioritised or resourced, creating an illusion of learning without actual change.
  • Failing to notify affected users about AI incidents that may have affected decisions or outcomes that concern them — transparency is an ethical obligation, not only a regulatory one.

5. Signals of Success

  • All significant AI incidents trigger a post-incident review, with written records of root cause analysis and improvement actions that are tracked to completion.
  • Mean time to detect and mean time to resolve AI incidents are measured and decreasing over time, reflecting improving monitoring and response capability.
  • The team has a library of incident runbooks that is used and updated, enabling effective response to recurring incident types without starting from scratch each time.
  • Affected users are notified of AI incidents that may have affected them, in accordance with documented communication protocols, within defined timelines.
  • Incident metrics are reviewed at a team level regularly, informing reliability improvement investments and demonstrating the team's commitment to operational excellence.