AI Operations Monitoring & Incident Runbook

Quick cheat-sheet

When an AI alert fires: 1) Validate the alert. 2) Triage data, model, and infra. 3) If user impact is high, apply safe mitigation (fall back, restrict, or revert). 4) Run root-cause checks, collect artifacts, and record the incident. 5) Follow the retrain or rollback playbook and run a post-incident learning review.

Purpose and scope

This runbook helps teams keep production ML services reliable, safe, and useful. It covers what to monitor, practical detection checks, triage steps, short-term mitigation, retrain criteria, governance and roles, communication templates, and a post-incident learning template. Use it for classification, regression, ranking, recommendation, and other prediction services.

Primary monitoring signals (what to watch)

• Model performance: primary business metric(s) and technical metrics (accuracy, F1, AUC, RMSE). Monitor rolling-window baselines and % change relative to recent history.
• Prediction quality proxies: calibration, confidence/entropy, top-k distribution changes.
• Input data distribution: per-feature distributions, multivariate drift tests, Population Stability Index (PSI), KL divergence or Earth Mover's Distance across rolling windows.
• Label quality & delay: label rate changes, label distribution drift, labeling backlog or label noise spikes.
• Data completeness: missing-field rates, schema changes, new categorical values, cardinality spikes.
• Latency & availability: p95/p99 latency, error rates, timeouts, throttling events.
• Resource and infra: CPU/GPU usage, memory, disk, node restarts, network error patterns.
• Business signals: conversion rates, fraud rate, customer complaints, support tickets tied to model outputs.

Example alert thresholds (starting points; tune to your context)

• Primary metric drop >5–10% relative to rolling 28-day average -> Warning (investigate); >15% -> Critical (mitigate).
• PSI > 0.2 for a major feature or KL divergence > 0.1 -> Investigate for data shift.
• Prediction confidence median falls by >20% -> Investigate calibration or input change.
• Missing-field rate > 2% (or sudden doubling) -> Investigate data pipeline.
• Latency p95 exceeds SLA by 30% or error rate >1% absolute increase -> Alert SRE and ML ops.

Document tuned thresholds for each model in the model registry. Use tiered thresholds (info/warn/critical) and require human confirmation for critical automated actions unless explicitly approved.

Triage checklist (first 30–60 minutes)

1. Validate the alert
   • Confirm the alert is not a duplicate or known maintenance.
   • Check instrumentation: are metric pipelines healthy? (ingestion lag, aggregator errors)
2. Assess impact
   • Which customers/users are affected? What product flows? Estimate scale and business risk.
3. Gather artifacts
   • Save snapshots: model version, serving config, recent input batches, a representative sample of predictions, logs, and traces.
4. Quick checks
   • Compare current input feature distributions vs baseline.
   • Check for recent code/config/data deployments in the last 24–72 hours.
   • Verify feature computation correctness and upstream pipeline health.
   • Look for label-shift or delayed labeling that might mislead metric signals.

Human‑in‑loop safety checklist

• Pause any automated remediation that would retrain or update models without human approval.
• Sample outputs and review with a domain SME or product owner before broad changes.
• Enable additional logging and sampling to preserve evidence for root-cause analysis.
• If bias/privacy concerns arise, notify Compliance and Data Ethics owners immediately and quarantine suspect data.

Short-term mitigation options (choose according to impact)

1. Fail open / degrade gracefully — return a safe default or business-rule result while preserving functionality.
2. Serve a previous stable model version — rollback to last known-good model (ensure config compatibility).
3. Restrict inputs — reject or sanitize out-of-range or new categorical values.
4. Apply thresholding — only surface predictions above a confidence threshold and route uncertain cases to human review.
5. Throttling / circuit breaker — reduce model load, route to cached responses, or temporarily disable non-critical features.

Apply the least disruptive mitigation that reduces user harm and allows time for root-cause analysis.

Incident escalation and roles

Key roles:

• Model Owner — accountable for model behavior and business impact.
• ML Engineer — diagnostics, retrain execution, model packaging.
• Data Engineer — data pipelines, schema, and storage checks.
• SRE/Platform — availability, latency, and infra mitigations.
• Product Owner/Business Stakeholder — user impact assessment and communications.
• Compliance/Privacy — engaged if personal data, bias, or regulatory risk detected.

Escalation matrix (example):

1. Alert -> On-call ML Engineer (15 min)
2. If not resolved or business impact significant -> Model Owner and SRE (30–60 min)
3. If user-facing harm or legal/compliance risk -> Product & Compliance (60–120 min)

Retrain decision and playbook

Only retrain when a clear, diagnosed problem with the training distribution or model has been identified and retraining is expected to improve the observed failure mode. Avoid blind retraining on recent data without handling label quality, covariate shifts, or sampling bias.

1. Decision criteria
   • Consistent performance regression on production data that matches a plausible training-data gap.
   • Acceptable label availability and quality for retraining.
   • Business owner approval and rollback plan in place.
2. Retrain steps
   1. Freeze deployment and mark incident in the model registry.
   2. Assemble training dataset: include stratified samples of the new inputs, ensure label hygiene, and apply pre-approved preprocessing.
   3. Run offline validation: offline holdout, backtesting, and targeted tests for the detected failure (e.g., performance on impacted segment).
   4. Run fairness, explainability, and privacy checks on candidate model.
   5. Deploy candidate to canary or shadow traffic with monitoring for the same signals.
   6. If canary looks good, follow incremental rollout with rollback triggers defined.

Rollback procedure

1. Identify last known-good model artifact and serving configuration.
2. Validate artifact compatibility with current feature pipeline and API contract.
3. Deploy rollback in a staged way (canary then gradual) where possible and monitor the same metrics for recovery.
4. Keep the problematic model artifact in quarantine for post-incident analysis.

Post-incident learning template

Capture the incident with the following fields and store in the model registry or incident tracker.

1. Incident ID & title
2. Summary — short description of what happened and user impact.
3. Timeline — detection time, actions taken, mitigation applied, resolution time.
4. Root cause hypothesis — data, model, infra, config, or process error.
5. Detection gaps — what signals were missing or noisy?
6. Corrective actions — immediate and long-term actions (with owners and due dates).
7. Validation to prevent recurrence — new tests, new monitoring metrics, required sign-offs.
8. Artifacts — links to saved snapshots, logs, model artifacts, experiment runs.
9. Lessons learned — what changed in process, policy, or architecture?

Governance, auditing, and record-keeping

• Record every deployment, model version, and configuration change in the model registry with immutable metadata.
• Store incident artifacts, investigation notes, and post-incident reviews against the model record.
• Require a documented acceptance test and fairness/privacy assessment before automated retrain or auto‑deploy is enabled.
• Define who can approve rollbacks, retrains, and changes to alert thresholds.

Communication templates

Initial incident notification (short)

Subject: [INCIDENT] Model <model-name> — potential degradation detected
Body: We detected an alert for <model-name> at <time> indicating <brief symptom>. Impact: <estimated impact>. Initial action: <mitigation or triage step>. Owner: <Model Owner>. We will provide updates every <interval> or when status changes.

Status update

Subject: [UPDATE] Model <model-name> — <status>
Body: Current status <investigating/mitigated/resolved>. Summary of actions taken: <list>. Next steps: <list>. ETA: <time>.

Resolution and post-incident

Subject: [RESOLVED] Model <model-name> — incident closed
Body: Summary of root cause, corrective actions, and owner for long‑term fixes. Link to post‑incident report: <url>.

Operational best practices and notes

• Prefer multiple complementary signals rather than single-metric automation triggers.
• Keep a human-in-the-loop for high-impact models and rare failure modes.
• Use shadow testing & canaries to evaluate candidate models before full rollout.
• Automate rollbacks only with well-tested safety nets and human approval for high-risk contexts.
• Regularly review thresholds and monitoring coverage as the product and data evolve.

Useful references & checklists

• Model registry entry for <model-name> (link)
• Incident tracker template (link)
• Monitoring dashboard (link)
• Canary deployment playbook (link)

How to use this runbook

Embed this runbook alongside model registry entries and monitoring dashboards. Make the triage checklist and communication templates easily accessible to on-call personnel. Periodically run simulated incidents (game days) to test detection, triage, and communication processes.