MLOps Oncall Runbook & Incident Playbook

Overview

This runbook helps oncall engineers, data scientists, product owners, and SREs detect and respond to production model incidents quickly and safely. It focuses on repeatable actions, clear roles, minimum-viable containment, rollback and safe-mode options, evidence collection for root cause, communication templates, and criteria that trigger model retraining or quarantine.

Who should use this playbook

This is for teams operating models in production (inference or decisioning). Typical users: oncall engineers, ML engineers, data engineers, data scientists, SREs, product owners, compliance/security reps, and support leads.

Quick-Action Checklist (first 15 minutes)

1. Confirm incident: note time, symptoms, and which service, endpoint, or dataset is affected.
2. Set incident channel (Slack/MS Teams) and incident owner (single point of coordination).
3. Place system into safe-mode if available (feature-flag to route to fallbacks or disable model decisions).
4. Notify stakeholders via incident template (see Communication Templates).
5. Begin collecting ephemeral evidence: recent inference logs, input batches, feature distributions, model confidence scores, monitoring alerts, and relevant traces.

Severity Levels and Response Targets

Use severity to decide escalation and time targets. Adjust labels to match your org’s incident taxonomy.

• Severity 1 — Critical: model causes safety, legal, or major revenue loss, or system unrecoverable. Target: respond < 15 min, mitigate < 60 min.
• Severity 2 — High: materially degraded performance affecting many users or key metrics. Target: respond < 30 min, mitigate < 4 hours.
• Severity 3 — Medium: noticeable but limited impact or localized errors. Target: respond < 4 hours, mitigate < 1 business day.
• Severity 4 — Low: minor or cosmetic issues, investigations, or false positives. Handle in normal sprint cadence.

Roles & Responsibilities

• Incident Owner: coordinates actions, communications, and decisions.
• Oncall Engineer / ML Engineer: investigates telemetry, applies containment or rollback, runs diagnostics.
• Data Scientist: analyzes model outputs, drift signals, and data quality issues.
• Data Engineer: validates data pipelines and upstream dataset integrity.
• Product/SLA Owner: assesses user impact and prioritizes mitigation vs. rollback.
• Compliance/Security: engaged for incidents with regulatory or PII risk.

Triage Checklist

1. Identify scope: which endpoints, customers, regions, or user cohorts are affected?
2. Determine symptoms: increased error-rate, latency, low confidence, distribution drift, label mismatch, or downstream business metric change?
3. Check recent deployments, config changes, feature-store updates, or infra events.
4. Collect telemetry snapshot (see Required Evidence).
5. Decide immediate action: contain (feature-flag/safe-mode), rollback model, or continue investigation.

Immediate Containment Options

Choose the least-disruptive effective containment:

• Safe-mode / fallback: route requests to a previous stable model, a heuristic rule set, or a cached response.
• Feature-flag disable: turn off risky features or entire model-based decision path.
• Rate-limit / circuit-breaker: throttle traffic to inference service to reduce impact while investigating.
• Quarantine inputs: temporarily drop suspicious input sources or batches (with audit logs).

Rollback & Redeployment Procedures

1. Confirm rollback candidate (previous model version tag, docker image, or saved model artifact).
2. Validate rollback in a smoke environment if possible (canary or pre-prod) for a short window.
3. Execute rollback with clear timestamped record and change ticket.
4. Monitor KPIs and regressions closely for at least the next 24 hours.
5. If rollback fails or is infeasible, escalate to Product/Leadership for business decision (accept degraded quality vs. downtime).

Data & Evidence to Collect (minimum)

• Inference logs: input features, model outputs, confidence scores, timestamps, user/customer id (masked if PII).
• Feature distributions and recent stats vs. baseline (means, variances, missingness).
• Model version and checksum, container/image tag, commit hash, and deployment metadata.
• Recent training data snapshot and last retrain timestamp.
• Upstream pipeline logs and schema change logs.
• Alerts, traces, and host/infra metrics (CPU, memory, latency, error rates).
• Relevant business metric deltas (conversion, revenue impact, error counts).

Retraining Triggers & Rules of Thumb

Use quantitative thresholds where possible; combine automated signals with human review:

• Feature drift above threshold (e.g., KL divergence or population stability index) sustained for N windows.
• Performance drop on labeled validation or shadow data beyond X% of baseline for Y time.
• Data pipeline corruption or label schema changes that invalidate features.
• Regulatory or safety issue indicating retraining with corrected labels is required.

When retraining is needed, create an isolated experiment, validate on holdout and production-similar datasets, and stage rollout via canary before full deployment.

Communication Templates

Incident Start (use in incident channel)

Subject: [INCIDENT] Model inference anomaly — {model-name} — {severity}

Body: Owner: {name}. Detected: {time}. Symptoms: {brief description}. Scope: {endpoints/customers/regions}. Immediate action: {safe-mode/rollback/none}. Next update: {time}.

Incident Update

Time: {time}. Action taken: {actions}. Evidence: {links to logs/dashboards}. Impact: {current impact statement}. ETA for resolution: {estimate}.

Resolution Notice

Time resolved: {time}. Root cause (initial): {short summary}. Mitigation applied: {rollback/retrained/patch}. Postmortem planned: {date/time}.

Postmortem Template

Use this template within 72 hours of resolution. Record facts, timeline, contributing causes, actions, and owners.

1. Title and incident ID
2. Summary of impact and duration
3. Timeline of key events (timestamps, actions, decisions)
4. Root cause analysis (data, model, infra, process, human)
5. Corrective actions (short-term and long-term) and owners with due dates
6. Monitoring & alerting changes to prevent recurrence
7. Lessons learned and suggested process updates

Common Failure Modes & Quick Diagnostics

• Data drift: check feature histograms and missingness; compare to baseline window.
• Label skew/feedback loop: inspect recent labels and sampling changes.
• Deployment/config mismatch: verify model artifact checksum, environment variables, and feature transformations.
• Scaling/infra: check latency, error rates, autoscaler events, and resource exhaustion.
• Third-party dependency failure: check upstream APIs and data provider liveness.

Monitoring & Alerting Recommendations

• Capture both system metrics (latency, error rate) and model metrics (confidence distributions, top-K predictions, coverage, input feature drift).
• Define anomaly detectors with cooldown windows and combine signals to reduce false positives.
• Instrument canary deployments and set automated rollback triggers for high-severity regressions.

Runbook Maintenance

Keep this playbook versioned with the model. After each incident, update the runbook with what worked, timelines, new thresholds, and any new telemetry queries. Link the runbook to the model artifact and deployment pipeline metadata.

Useful Snippets & Links

• Telemetry queries: include example queries in your monitoring system to capture input distributions, recent predictions, and confidence bands.
• Model artifact identifiers: always record model tag, commit hash, and build pipeline id in incident notes.
• Evidence storage: link to centralized incident evidence (logs, dashboards, sample inputs) so reviewers can reproduce findings.

Appendix — Minimum Evidence Checklist

• Model version and deployment metadata
• Inference logs for the incident window
• Feature distribution snapshots (current vs baseline)
• Alert and error traces
• Recent code or config deployments
• Business metric deltas