Model Deployment & Canary Runbook

Purpose & Scope

This runbook describes the validated, repeatable procedure for deploying machine learning models into production with a canary rollout, monitoring checks, and safe rollback procedures. It is written for engineers, SREs, ML engineers, and product owners responsible for operating models that serve live traffic. Use this document for routine model replacements, major model upgrades, and emergency fixes.

Roles & Contacts

• Deployment Owner: Responsible for running the deployment and initial validation.
• SRE / Platform Lead: Manages traffic routing, platform-level rollback, and monitoring integration.
• ML Owner / Reviewer: Validates model outputs, fairness checks, and business impact.
• Incident Communicator: Sends updates to stakeholders and coordinates postmortem.

Quick Run Instructions (High-level)

1. Run the Pre-deployment Checklist and obtain approvals.
2. Deploy candidate model to canary endpoint and shift a small traffic slice.
3. Monitor evaluation metrics during the configured window using the Canary Plan.
4. If metrics remain within acceptance thresholds, continue phased traffic shift to 100% or as planned.
5. If rollback criteria trigger, execute the rollback procedure and open an incident.
6. Run Post-deployment validation and document results; schedule retraining if triggers fire.

Pre-deployment Checklist

All items must be completed before initiating the canary rollout.

• Model validation: Unit tests, integration tests, and end-to-end inference tests pass. Include deterministic test vectors and edge cases.
• Performance & latency: Latency and throughput at target QPS measured in staging under realistic load; p95/p99 within SLOs.
• Accuracy & metrics: Candidate model’s key metrics (e.g., accuracy, F1, AUC, RMSE, business KPIs) compared against baseline with confidence intervals and pre-agreed delta thresholds.
• Fairness & data leakage: Run bias checks, holdout fairness scans, and verify no training-serving skew or PII leakage.
• Canary config ready: Traffic routing rules, feature flags, and experiment configuration reviewed and stored in version control.
• Monitoring & alerts: Dashboards, alerts, and log pipelines set up for the new model version (metrics, errors, data distributions).
• Rollback plan: Verified procedure to revert to previous model and expected recovery time documented.
• Data retention & compliance: Confirm model artifacts, logs, and telemetry meet retention and privacy policies.
• Approvals & change control: Deployment approved by required stakeholders (list approvals and timestamps).

Canary Plan

Use a phased traffic shift and an explicit evaluation window. Example rollout plan:

1. Canary slice: 1% of traffic for 30 minutes
2. If steady: increase to 5% for 60 minutes
3. If steady: increase to 25% for 2 hours
4. Final: ramp to 100% (or a business-approved percentage) with monitoring for 24 hours

Evaluation guidance:

• Primary metrics: model-specific business metrics (e.g., conversion rate lift, false positive rate, RMSE). Compare canary vs baseline.
• Secondary metrics: latency (p50/p95/p99), error rates (5xx/4xx), resource usage (CPU/GPU / memory), and downstream system KPIs.
• Statistical checks: Use pre-defined significance thresholds or non-inferiority margins. For small canary slices, prefer sequential monitoring methods and guard against false positives.
• Data sampling: Ensure canary traffic is representative. If traffic is heterogeneous, create targeted slices (e.g., new vs returning users, geographies).
• Evaluation window: Long enough to capture periodic effects (e.g., hourly patterns) and short enough to limit exposure; document rationale for chosen window.

Monitoring Widgets and Alert Thresholds

Suggested dashboard widgets and baseline alert thresholds. Tune for your service and risk profile.

• Model quality widget: key metric time series (canary vs baseline). Alert when delta exceeds configured margin. Severity: P1 if > critical delta, P2 if > warning delta.
• Latency & error widget: p50/p95/p99 latency and 5xx error rate. Alert if p95 increases by > 20% or 5xx rate > 0.5% above baseline.
• Throughput & resource usage: QPS, GPU/CPU, memory. Alert on resource saturation or unexpected increases that suggest runaway behavior.
• Input distribution monitors: feature histograms, missing-value rates, and schema changes. Alert on drift beyond thresholds (e.g., KL divergence or population shift metric above threshold).
• Bias & fairness checks: subgroup performance metrics. Alert if subgroup degradation exceeds allowed delta.

Alerting policy: map alerts to owners, required response times, and escalation path. Provide runbook links in each alert message.

Rollback Criteria & Procedures

Rollback should occur automatically for clearly defined critical failures and manually for other conditions. Predefine triggers that force automatic rollback (and test them):

• Critical error rate spike (e.g., 5xx > 1% and sustained for 5 minutes)
• Severe latency degradation outside SLOs for > 10 minutes
• Model quality metric drops beyond critical delta in evaluation window
• Schema or data-pipeline break detected (feature missing or corrupted)

Rollback steps (example):

1. Automatic rollback: Monitoring automation triggers traffic switch back to previous model version and notifies the on-call team.
2. Manual rollback:
   1. Notify stakeholders: use Incident template (below).
   2. Invoke platform rollback command (example): kubectl apply -f service-deployment-previous.yaml or helm rollback .
   3. Verify routing: confirm traffic is back to baseline and health checks pass.
   4. Monitor for stabilization (30 minutes minimum) and capture logs and telemetry for root cause analysis.
3. Data and state considerations: If the new model wrote new outputs or triggered downstream changes, ensure compensating actions or data reconciliation is available before resuming traffic to the candidate.
4. Model registry revert: Mark the candidate as failed in the model registry and tag the baseline as active. Record reasons and evidence.

Post-deployment Validation & Retraining Triggers

After the model is fully deployed (or rolled back), run validation and decide on next steps.

• Smoke checks: Run a curated set of inference tests across critical segments and edge cases.
• Holdout evaluation: Re-evaluate on reserved holdout data (if available) and compare expected metrics to live metrics.
• Drift detection: Continuous data-distribution and performance monitoring. Trigger retraining when drift metrics exceed thresholds or when business KPIs degrade persistently.
• Retraining policy: Decide whether to start an automated retraining pipeline or create a ticket for manual model refresh. Record the trigger condition that led to retraining.

Communication Templates

Use these templates to speed stakeholder updates during a rollout or incident.

Slack / IM - Canary Started

Subject: Canary started for model {model-name}
Message: Canary rollout begun at {time}. Traffic: {1%}. Watch metrics: . Owners: {deployment owner}, {SRE}.

Slack / IM - Canary Success

Subject: Canary OK — proceeding to next phase
Message: Canary evaluation passed for window {window}. Increasing traffic to {next-slice}. No critical alerts. Next checkpoint at {time}.

Slack / IM - Canary Failure / Rollback

Subject: Rollback initiated for model {model-name}
Message: Canary failed due to {short reason}. Rolling back to {baseline-version} at {time}. Incident channel: {link}. Owners notified: {names}.

Email — Postmortem Request

Subject: Postmortem: Deployment {model-name} {version} — {success|rollback}
Body: Provide timeline, metrics that triggered action, logs, root-cause hypothesis, and next steps. Assign owner for follow-up.

Postmortem & Continuous Improvement

After any non-trivial rollout (success with issues or rollback), do a short postmortem within 3 business days. Capture:

• Timeline of events
• Contributing factors
• Mitigations and remediation actions
• Actions to improve the runbook or tooling

Appendix: Useful Commands & Examples

Examples vary by platform. Replace placeholders before running.

• Helm rollback: helm rollback my-release 2
• Kubernetes set image: kubectl set image deployment/my-svc my-container=my-image:tag --record
• Traffic split (example for Istio virtualservice): update weight fields for versions.
• Model registry tag: model-registry tag {model-id} --set active=true
• Query example for comparison: SELECT COUNT(*) FROM inferences WHERE model_version IN ('canary','baseline') GROUP BY model_version, label;

Safety Notes & Limits

This runbook reduces risk but does not remove it. Do not use it to justify deploying models in high-risk domains (health, legal, safety-critical) without domain expert review and additional controls. Always follow organizational compliance procedures.

Quick Reference Checklist (Printable)

• [ ] Approvals obtained
• [ ] Pre-deployment tests passed
• [ ] Canary config deployed
• [ ] Dashboards & alerts active
• [ ] Rollback procedure verified
• [ ] Communication channel ready
• [ ] Post-deployment validation scheduled