MLOps Runbook: Deployment, Monitoring & Retraining

Purpose

This playbook helps teams reliably deploy machine learning models to production, detect and respond to degradation, retrain safely, control costs, and hand off operational responsibility to SRE or platform teams. It focuses on pragmatic patterns you can adapt to your environment rather than prescriptive architecture.

Scope

Covers deployment patterns, testing and CI/CD checkpoints, observability (metrics, logs, traces), drift detection approaches, retraining triggers & safe retraining pipelines, versioning & rollback, incident runbooks, and SRE handoff checklists.

Preconditions & Roles

• Stakeholders identified: model owner, data owner, ML engineer, SRE/site owner, product owner, compliance officer.
• Artifact registry available (model registry, container registry, or object storage) and clear artifact immutability policy.
• CI/CD pipeline or orchestration tooling (e.g., GitOps, Jenkins, GitHub Actions, ArgoCD) to automate build, test, and deploy steps.

Deployment Patterns & Checklist

Choose a pattern that matches risk and traffic: rolling, blue/green, canary, shadow (mirroring), or A/B. For model updates prefer incremental traffic shifts where possible.

Deployment checklist (adapt and require signoffs where needed)

• Package model as immutable artifact (model file + metadata + environment spec).
• Build reproducible container or serverless image with pinned dependencies.
• Run unit tests and model-specific tests (sanity checks, consistency tests, input/output schema validation).
• Run integration tests against a staging dataset and mock services.
• Perform offline validation: baseline metrics vs new model on holdout data.
• Smoke test in staging environment with synthetic and representative inputs.
• Deploy with conservative traffic shift (canary) and validate runtime metrics before full rollout.
• Document expected behaviour, known limitations, and rollback criteria in the release notes.

Observability: What to Measure

Instrument models and surrounding services so engineers can detect behavioural change quickly.

Essential telemetry

• Service health: latency p50/p95/p99, request rate, error rate.
• Model performance proxies: prediction distribution, confidence/score distributions, top-k predictions frequency.
• Business KPIs where available: conversion rate, revenue per request, false positive/negative rates (if labeled feedback exists).
• Data metrics: input feature distributions, nulls/missingness, schema changes.
• Resource & cost metrics: CPU/GPU utilization, memory, inference cost per request.

Example alert examples (use these as starting points and tune for your system):

• Latency increase: sustained p95 latency > X ms above baseline for Y minutes.
• Error spike: request error rate exceeds normal by a factor of Z.
• Prediction drift: significant change in prediction distribution or feature means (statistical test threshold) over N batches.
• Business impact trigger: drop in conversion or other downstream KPI beyond tolerance window.

Drift Detection Recipes

Combine statistical tests with business-aware checks for robust drift detection.

1. Data drift (unsupervised): compare recent feature distributions to training baseline using tests such as Population Stability Index, KL divergence, or two-sample tests. Monitor feature-level and joint distributions where feasible.
2. Label drift (supervised where feedback exists): track label distribution and conditional model performance on newly labeled data.
3. Prediction drift: monitor changes in predicted classes, confidence calibration, or class imbalance over time.
4. Concept drift detection: monitor performance on recent labeled samples or use proxy labels when true labels lag (e.g., delayed business outcomes).
5. Practical recipe: set short-term (hour/day) and medium-term (week) windows, compute drift metrics, and require corroborating signals (e.g., drift + latency change + KPI change) before triggering expensive actions.

Retraining Triggers & Strategy

Retraining should be driven by measurable triggers and guarded by validation stages.

Common triggers

• Performance threshold: model metric falls below an agreed SLO (e.g., F1, RMSE) on a validation stream.
• Detected drift: sustained data or prediction drift beyond threshold plus business impact signal.
• Scheduled cadence: periodic retraining (weekly/monthly) for non-stationary domains or where labeling lags.
• Regulatory or business changes: new feature sets, label schema changes, or product shifts.

Safe retraining process

1. Define training dataset selection strategy and version its data snapshot.
2. Run automated training in CI with deterministic seeds and dependency pinning.
3. Run a full validation suite including holdout tests, fairness checks, and adversarial or edge-case tests if relevant.
4. Compare candidate model vs production baseline using pre-defined acceptance criteria (statistical significance tests where helpful).
5. Stage candidate in shadow mode (mirror traffic without affecting outcomes) for a validation period.
6. Promote via canary to small traffic with strict monitoring and rollback windows.

Versioning, Registry & Rollback Procedures

• Use a model registry that records model artifact, training data snapshot, evaluation reports, hyperparameters, and environment spec.
• Adopt clear version semantics (major.minor.patch) and tag each production deployment with the registry version and commit hash.
• Keep previous production artifacts available for fast rollback and record any schema migrations required by new models.
• Rollback criteria should be explicit (e.g., drop in KPI beyond X% for Y minutes) and executable via automation where possible.

Cost Controls & Efficient Inference

Modelling choices often drive cost. Include cost-aware decisions in the runbook.

• Use autoscaling, batching, and request-level caching to reduce wasteful compute.
• Consider model optimization: quantization, pruning, distillation for latency/cost-sensitive paths.
• Route high-volume, low-risk traffic to cheaper models or lower-fidelity modes when acceptable.
• Monitor inference cost per request and set budget alerts to avoid runaway spend.

Incident Response: Runbook Template

When an alert fires, follow a structured incident process to reduce time-to-resolution and preserve evidence.

Quick incident triage

1. Record incident ID, timestamp, and alert details in the incident tracker.
2. Notify stakeholders (on-call ML engineer, SRE, product owner).
3. Assess severity: user-visible impact, business metric degradation, or internal-only issue.
4. If user-impacting, consider immediate traffic diversion to previous stable model or degrade gracefully to a deterministic fallback.

Investigation checklist

• Examine service health metrics (latency, errors) and logs for exceptions.
• Check recent deployments and configuration changes (including infra autoscaling events).
• Review recent data distributions for sudden changes or upstream schema changes.
• If retraining or shadow experiments are running, correlate timing with the incident.

Resolution & follow-up

• Document root cause, steps taken, timeline, and mitigations in the incident postmortem.
• Apply fixes (rollback, hotfix, data correction) and verify through targeted tests.
• Create follow-up work items for preventing recurrence (monitoring gaps, automation, tests).

SRE Handoff Checklist

• Provide runbooks and runbook playbooks (this document) linked from the service runbook index.
• Ensure dashboards and alerts are accessible and documented with clear ownership.
• Confirm rollback automation is in place and tested periodically.
• Agree on escalation paths and RTO/RPO expectations for model-related incidents.

Templates & Useful Artifacts

Include or link the following artifacts in your project repo or registry:

• Deployment checklist (editable checklist for each release).
• Incident form (fields: incident ID, start time, alerts, stakeholders, remediation steps, final state).
• Model metadata template for registry entries (training data snapshot, evaluation metrics, owners, lineage).
• Retraining pipeline definition and validation report template.

Practical Tips & Common Pitfalls

• Do not rely only on raw accuracy; monitor downstream business signals where possible.
• Avoid silent failures by adding health-check endpoints that validate end-to-end inference with lightweight tests.
• Beware of data leakage in online validation — use realistic held-out streams where possible.
• Automate rollbacks and tests; manual-only procedures increase MTTR and risk.

Next Steps & Adaptation

Adapt thresholds, tests, and cadence to your domain. Start small: instrument a few key metrics, define one retraining trigger, and iterate. Use shadow deployments to validate model behaviour before risking user-facing effects.