MLOps Deployment Checklist & Playbook

Welcome — why this playbook matters

Deploying a model is only the beginning. Models can degrade, change data behavior, or create business risk if deployed without careful packaging, validation, monitoring, and rollback plans. This playbook gives a pragmatic checklist and reusable templates so teams can deliver consistent, observable, and recoverable model deployments.

How to use this playbook

Use the checklists as a pre-deployment gate, the monitoring and alerting guidance as your runbook, and the templates as starting points you must adapt to your stack and governance. Preserve provenance, make small incremental rollouts, and define clear human-in-the-loop steps for risky changes.

Pre-deployment checklist (must-haves)

• Model artifact: Versioned artifact (container image, model file), checksum, and model card that includes intended purpose, data sources, training date, evaluation metrics, and known limitations.
• Data contracts: Input schema, feature definitions, and acceptable value ranges. Include sentinel checks for missing or extreme values.
• Reproducibility: Commit hash for training code, environment spec (requirements.txt/conda lock), random seeds, and training dataset identifier(s).
• Model registry: A single source of truth with model versions, metadata, and access controls.
• Security & privacy: Secrets management, encryption at rest and in transit, and a data-minimization review for any PII or sensitive inputs/outputs.
• Acceptance criteria: Clear quantitative gates (performance, fairness metrics, latency) and business KPIs required to allow deployment.

CI/CD pipeline — tests and automation

Automate build, test, and promotion. Typical stages:

1. Build: create container image/artifact and compute artifact checksum.
2. Unit tests: model utilities, feature transforms, and edge-case handling.
3. Integration tests: model inference end-to-end using canned inputs, schema validation, and dependency checks.
4. Model quality tests: smoke evaluation against a holdout or synthetic dataset with pre-defined pass/fail thresholds.
5. Security/scan: container image vulnerability scan, secret leakage scan.
6. Promote: push to registry and tag as staging/production candidates when gates pass.

Example: minimal CI snippet (GitHub Actions-style)

<!-- Example: simplified for illustration; adapt to your runner and registry -->
name: ci
on: [push]
jobs:
  build-and-test:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v2
      - name: Build image
        run: docker build -t registry.example.com/my-model:${{ github.sha }} .
      - name: Run unit tests
        run: pytest tests/unit
      - name: Run model smoke test
        run: python tests/smoke_inference.py --model ./artifacts/model.pkl --threshold 0.7
      - name: Push image
        if: success()
        run: docker push registry.example.com/my-model:${{ github.sha }}
  

Deployment strategies

Deploy incrementally with the ability to observe and revert quickly.

• Shadow: Route live traffic copies to the new model for observation without affecting responses.
• Canary: Send a small percentage of live traffic to the new model, measure business and technical metrics, then expand if healthy.
• Blue-Green: Deploy new model to parallel environment and switch traffic atomically when ready.

Example: Kubernetes Deployment (simplified)

<!-- Deployment: two replicas; service and deployment combine with your traffic-split mechanism -->
apiVersion: apps/v1
kind: Deployment
metadata:
  name: my-model-v1
spec:
  replicas: 2
  selector:
    matchLabels:
      app: my-model
      version: v1
  template:
    metadata:
      labels:
        app: my-model
        version: v1
    spec:
      containers:
      - name: model
        image: registry.example.com/my-model:v1
        ports:
        - containerPort: 8080

For canaries, use your service mesh or traffic controller (Istio, Linkerd, Ambassador, or cloud load balancer) to split traffic between versions and to perform gradual ramps.

Runtime monitoring & observability

Monitor both technical and model quality metrics. Capture signals at inference time and correlate with business KPIs.

Essential metrics

• Technical: availability, CPU/RAM, latency percentiles (p50/p95/p99), throughput, error rate (4xx/5xx).
• Model quality: prediction distribution, class balance, confidence histogram, online surrogate metrics (if labels delayed), and sampled ground-truth accuracy when available.
• Data drift: feature distribution shifts (e.g., PSI), schema changes, null rate increases, unseen categorical values.
• Business: downstream KPIs such as conversion rate, revenue per session, false positive/negative counts tied to cost.

Logging & traces

Log inputs (respecting privacy), model outputs, model version, request ID, timestamps, and trace ids to link logs with traces and business events.

Drift detection, retrain triggers, and governance

Design retrain triggers that balance automation and human oversight.

• Automated triggers: sustained metric breach over N windows (e.g., PSI > 0.2 for 3 days) or a sudden drop in business KPI correlated to model usage.
• Human-in-the-loop: require data scientists or model owners to review drift alerts before initiating production retraining when the model affects high-risk outcomes.
• Retrain workflow: snapshot production data, run training with same code, validate against gates, and push candidate to staging shadow for A/B testing.
• Bias & fairness checks: run automated fairness tests on new candidates and compare with baseline. Fail fast if regressions appear on protected groups.

Alerting thresholds & runbook (example)

Define actionable alerts and explicit runbook steps.

• Latency p95 > 1s for 10 minutes: Action — scale replicas, check recent deploy, rollback if persists.
• Prediction error rate increase > 3x baseline (if real-time labels available): Action — enable shadow & run regression tests; notify model owner.
• Feature PSI > 0.2 for two consecutive days: Action — investigate upstream data changes, assess retrain necessity.
• Resource exhaustion or OOM on pods: Action — autoscale, restart pods, and roll back if due to version change.

Each alert should map to a documented runbook that includes:
who to notify, initial triage steps, how to route traffic away from the model, how to revert to previous model version, and how to capture forensic logs.

Rollback & emergency steps

1. Open incident and notify stakeholders (SRE, model owner, product owner).
2. Stop or divert traffic to previous stable model (traffic split to 100% stable version), or scale down offending pods.
3. Mark model version as failed in registry and revoke any automatic promotions.
4. Collect logs, traces, and a dataset snapshot for postmortem.
5. Execute post-incident review and update tests, gates, and runbooks.

Governance, auditability & compliance

• Store full model lineage: training code, dataset ids, hyperparameters, and evaluation artifacts.
• Enable access controls and audit logging for who promoted or deployed a model and when.
• Document intended use and limitations in the model card and require sign-off for high-risk use cases.
• Where required, retain explainability artifacts and justification records for decisions driven by model outputs.

Minimal SLA template for a model

Adapt this template to your business needs and legal constraints.

Model Name: My Model
Service Level: Production inference endpoint
Availability: 99.5% monthly uptime (exclusions: scheduled maintenance)
Latency: p95 latency < 300ms under normal load
Prediction Quality: Holdout AUC > 0.82 or business KPI within ±5% of baseline
Data Retention: Inference logs retained for 90 days for auditing
Change Control: Any model with risk class >= 2 requires stakeholder approval and a documented roll-out plan
Incident Response: Critical failures acknowledged within 15 minutes; mitigation plan within 1 hour
  

Example: simple canary expansion plan

1. Deploy new model version to staging; run synthetic and shadow tests for 24 hours.
2. Canary: route 1% traffic for 1 hour; monitor defined metrics.
3. If no regressions, increase to 10% for 4 hours; reassess.
4. Increase to 50% for 24 hours with business KPI monitoring.
5. Full rollout if gates remain green; otherwise rollback to previous stable version and start investigation.

Quick deployment checklist (one-page)

• Artifact built & checksummed
• Unit & integration tests passed
• Model quality gates passed
• Model card and lineage recorded
• Shadow or canary strategy defined
• Monitoring dashboards present for all essential metrics
• Alerting rules and runbooks in place
• Rollback playbooks tested
• Access control & audit enabled

Next steps and tailoring

Customize thresholds, tests, and rollout percentages to your risk tolerance, data cadence, and business impact. Consider integrating with a model registry, observability platform, and feature store for stronger lineage and faster root cause analysis.

If you want this playbook as an interactive checklist that records deployments, incidents, and metric observations, consider enabling an interactive checklist and submission workflow linked to your model registry and alerting systems.

Appendix — resources and references

• Model cards and documentation templates
• Standard drift diagnostics: PSI, KS test, population stability, multivariate drift
• Canary & traffic-splitting patterns for popular service meshes and cloud providers