MLOps Deployment & Incident Runbook

Purpose & Scope

This runbook describes operational procedures for deploying machine-learning models, validating them in production, detecting and responding to incidents, and conducting post-incident reviews. It targets engineers, SREs, ML engineers, product owners, and on-call responders responsible for model-backed services. It does not replace security reviews, legal compliance checks, or high-risk domain approvals.

Roles & Responsibilities

• On-call responder: First-line investigation, containment, and communications.
• ML Engineer / Model Owner: Model-level diagnostics, rollback, redeploy, and retraining decisions.
• SRE / Platform Engineer: Traffic routing, infra rollback, logs/metrics access, debugging infra issues.
• Product / Business Owner: Impact assessment, stakeholder updates, business decisions (e.g., degrade features).
• Compliance / Security: Invoked when incidents involve data leakage, privacy, or regulatory exposure.

Pre-deployment Checks (Gate Checklist)

Run these checks before any production traffic is routed to a model version.

1. Data & Feature Checks
   • Run input schema validation on a representative sample. Confirm no required features missing or unexpected null rates.
   • Sanity-check feature distributions vs. recent production baseline (population shift/drift alert if > X% divergence). Example: >10% shift in categorical distribution or >3σ change in mean for critical numeric features.
2. Model Validation
   • Unit tests for prediction code and feature engineering pipelines pass.
   • End-to-end inference test with synthetic and real examples. Compare outputs with expected ranges and business constraints.
   • Explainability smoke test: run explainability on a small set and verify no unexpected attribution patterns (e.g., single feature dominance if not expected).
3. Performance & Capacity
   • Latency and memory benchmarks meet SLAs (e.g., p95 latency < 200ms for online models — adapt to your SLA).
   • Load-tested at expected concurrency levels.
4. Security & Privacy Confirm data handling, encryption, and access controls are in place. Verify no PII leakage in logs.
5. Observability
   • Metrics, logs, and traces instrumented: request counts, latency, error rate, model confidence distributions, input feature drift metrics, prediction distribution, and example request/response logging (scrubbed if PII).
   • Dashboards and alerts created for baseline and canary monitoring.

Deployment Steps (Canary First)

Use controlled rollouts to minimize blast radius.

1. Prepare Canary
   • Deploy new model to a canary pool or separate endpoint.
   • Ensure telemetry from canary is isolated and compared against production baseline.
2. Traffic Split
   • Start with a small percentage of traffic (e.g., 1-5%). Hold for observation window (e.g., 30–60 minutes depending on traffic volume).
   • Gradually increase in steps (5→20→50) only if metrics remain within acceptable ranges and no regressions observed.
3. Monitoring During Rollout
   • Continuously compare canary vs baseline on these metrics: error rate, latency p95/p99, model confidence distribution, key business KPIs (e.g., conversion rate), and feature drift indicators.
   • Use automatic canary analysis if available (statistical checks or A/B testing framework).
4. Promote or Abort
   • If canary passes for the full observation window and metrics are stable, promote to full rollout following incremental traffic steps.
   • If a threshold breach occurs, follow rollback criteria (next section).

Alert Thresholds (Examples & Guidance)

Thresholds below are starting examples. Tune them to your production characteristics.

• Error Rate: Alert if 5-minute error rate increases by >50% relative to baseline OR absolute error rate >1% (adjust per service).
• Latency: Alert if p95 latency increases by >2x baseline or exceeds SLA (e.g., >2s).
• Model Confidence Shift: Alert if mean confidence moves by >0.2 or if the proportion of low-confidence predictions increases by >50%.
• Prediction Distribution Drift: Alert on KL-divergence or PSI > threshold (e.g., PSI > 0.2 for a critical feature).
• Business KPI Degradation: Alert on significant negative movement in downstream KPI (conversion, approval rate) correlated with rollout window.

Incident Triage Flow

1. Detect & Acknowledge
   • On-call acknowledges alert in incident system and notifies stakeholders per communications template.
2. Initial Impact Assessment (5–15 min)
   • Scope: which endpoints, regions, user groups affected? Is this canary-only or global?
   • Severity: business impact, user-facing errors, data leakage risk.
   • Quick checks: recent deploys, infra incidents, rate spikes, telemetry anomalies.
3. Containment & Mitigation
   • If the issue is isolated to a model version: consider immediate traffic split rollback for that version.
   • If infra-related: escalate to SRE for infra rollback, scaling, or circuit breaker activation.
   • Mitigations include routing traffic to previous model, serving a cached response, or enabling a safe fallback rule.
4. Decide: Rollforward vs Rollback
   • Rollback if: clear model-related regression in key metrics, data leakage, or high-severity user impact. Follow rollback checklist below.
   • Rollforward (patch) if: root cause is limited, a hotfix is available and tested quickly, and risk of rollback is higher than patching.
5. Recover & Monitor
   • After action (rollback or patch), monitor for re-emergence for at least two observation windows and verify business KPIs recovered.
6. Post-Incident
   • Open an RCA ticket, record timeline, decisions, and owners. Schedule RCA meeting within 72 hours.

Rollback Criteria & Procedure

Rollback is the safe default for unexpected production model regressions.

1. Rollback Criteria
   • Canary or production metrics breach pre-defined thresholds and do not stabilize within the observation window.
   • Degradation of business KPIs that correlate with the deployment.
   • Evidence of data leakage or privacy gaps introduced by the release.
2. Rollback Procedure
   1. Notify stakeholders: short incident update using template (see Communications).
   2. Shift traffic away from failing model version to last known good version (traffic manager / service mesh / API gateway change).
   3. Confirm routing change took effect and verify baseline metrics returned.
   4. Lock the problematic model artifact and preserve logs and telemetry for RCA.
   5. Create a follow-up action item: who investigates, timeline for fix or retrain.

Post-Incident RCA Template

Use this structured RCA to capture facts and lessons.

1. Incident ID & Summary
2. Timeline — chronological events with timestamps (alerts, deploys, mitigation actions).
3. Impact — users, regions, business metrics affected, estimated duration.
4. Root Cause — technical and human contributing factors.
5. Immediate Fix — actions taken to restore service.
6. Long-term Remediation — code changes, monitoring improvements, process updates, training.
7. Owner & Due Dates — who will implement each remediation and when.
8. Verification Plan — how to validate the fix and prevent regression.

Stakeholder Communication Templates

Trim and use these templates for consistent, timely communication.

Testing & Exercises

Schedule recurring tabletop exercises for incidents and run periodic canary drills where a canary is intentionally introduced with a simulated regression to validate detection and rollback workflows.

Runbook Maintenance

Owner: Model Owner. Review cadence: every deployment pipeline change and quarterly. Update this runbook when onboarding new services, changing traffic routing mechanisms, or after every significant incident.

Appendix: Practical Examples & Commands

Include platform-specific commands and example dashboards in your team copy. Examples might include kubectl rollout undo commands, service-mesh traffic-split snippets, or monitoring queries for drift and confidence histograms.

Safety & Compliance Note

If there is any suspicion of data leakage, personal data exposure, or regulatory impact, pause public communications and escalate immediately to your compliance and security teams.