Model Deployment & Production Checklist

Purpose

This checklist helps teams reduce production incidents by standardizing checks, metrics, validation, and runbooks used when deploying machine learning models. Use it as a launch gate: every item should have an owner and evidence (test artifacts, links, screenshots, runbook IDs) before sign-off. Preserve the checklist record with the deployment for audit and continuous improvement.

How to use

For each checklist item: assign an owner, add evidence (link, artifact name, or short note), mark complete only when the acceptance criteria are met. If an item cannot be met, document the residual risk and mitigation, and require explicit approval from the responsible approver (product owner, SRE, security lead, or compliance officer as appropriate).

Pre-deploy validation

1. Unit & integration tests

   Acceptance: All model-related unit and integration tests pass in CI. Test coverage for new logic meets team standard. Provide CI run link and failing-test zero evidence.

   Evidence: CI job link, test report, coverage percentage.

2. Performance benchmarks

   Acceptance: Model inference latency and memory/CPU usage meet SLOs for target environment (batch/online). Include representative load test results for expected QPS.

   Evidence: Benchmark report, test dataset identifier, SLO values.

3. Model quality verification

   Acceptance: Evaluation metrics (accuracy, AUC, F1, RMSE, business KPI uplift) on a holdout or production-like dataset are within approved range compared to baseline. Provide test dataset and metric table.

4. Data contract assertions

   Acceptance: Feature schemas, data types, nullability, cardinality, and value ranges match expectations. Contracts for upstream producers and consumer contracts (e.g., prediction consumers) are documented and signed-off.

   Evidence: Schema version, contract changelog, automated assertion test results.

5. Bias and fairness checks

   Acceptance: Known bias checks completed, sensitive-attribute testing performed, and either no unacceptable bias found or mitigation plan documented and approved.

   Evidence: Fairness report, mitigation design, approver name.

6. Security & privacy review

   Acceptance: Secrets, keys, and PII handling verified. Threat model and data access reviewed by security. No sensitive data is leaked by model artifacts or logs.

   Evidence: Security review ticket, data flow diagram, encryption notes.

7. Explainability & artefacts

   Acceptance: Explainability artifacts required for this risk tier (feature importances, Shap summaries, counterfactual examples, or model card) are generated and stored where auditors and engineers can access them.

   Evidence: Links to model card, explainability dashboards, artifact storage path.

Deployment plan & rollout strategy

1. Release plan documented

   Acceptance: Deployment checklist includes target environments, versions, config flags, DB/migration steps (if any), and rollback steps. A named owner is assigned for deployment window.

2. Canary or staged rollout

   Acceptance: Canary percentage, duration, target slices (customers, regions, cohorts), and promotion criteria are defined. Automated traffic split and observability for the canary are configured.

   Evidence: Canary config, promotion script, traffic-routing rules.

3. Feature flag / kill switch

   Acceptance: Feature flag or runtime switch exists to quickly disable new behavior. Team members with permission to flip flag are identified and reachable during rollout.

Observability & monitoring

Ensure monitoring covers correctness, performance, and business impact. Instrumentation should be in place before traffic shifts to the new model.

• Latency — P50/P95/P99 and SLA alignment. Alert on sustained SLO violation.
• Error rate — Runtime exceptions and input processing errors with source-tagging.
• Prediction distribution — Key features and predicted label distribution vs. baseline.
• Input validation failures — Count and rate of contract/validator failures.
• Business KPIs — Primary business metric(s) the model influences (conversion, revenue per user, false positive cost).

Acceptance: Dashboards and alerts exist for each item, and alert thresholds and on-call run loops are defined.

Alerts, thresholds & runbooks

1. Alert definitions

   Acceptance: Alerts for SLO breaches, data drift spikes, prediction ratio shifts, and input validation errors are defined with clear severity levels and paging rules.

2. Rollback triggers

   Acceptance: Explicit rollback triggers are documented (e.g., >X% increase in error rate, significant drop in business KPI, model-serving OOMs). Each trigger ties to a required action and runbook.

   Evidence: Runbook IDs or links for rollback and investigation.

3. On-call runbook

   Acceptance: A readable runbook exists with steps for investigation, mitigation, rollback, and communication. Contacts for product, SRE, data engineering, and legal (if needed) are listed.

Post-deploy validation

1. Smoke & integration checks

   Acceptance: End-to-end smoke tests against production endpoints pass. Quick sampling of predictions shows expected behavior on representative inputs.

2. Performance & SLO verification

   Acceptance: Latency and error SLOs are verified under production traffic during the canary window.

3. A/B or shadow testing validation (if used)

   Acceptance: A/B metrics are collected and compared to baseline; no regression in primary business KPI or agreed statistical check passes.

Ongoing operation & lifecycle

• Drift detection — Feature distribution and label drift alerts configured. Define retraining cadence or triggers.
• Model versioning — Model artifact, training data snapshot, and code digest are stored and linked to the deployment.
• Retraining & rollback plan — Process for retraining, evaluation, and safe promotion is documented.

Governance, audit & compliance

• Access controls for model artifacts and prediction endpoints are enforced and audited.
• Logging of inputs/outputs preserved according to privacy standards and retention policies.
• Regulatory or contractual obligations are identified and signed off.

Quick sign-off checklist (minimum fields)

1. Checklist owner: ______________________
2. Deployment date/time: __________________
3. Pre-deploy tests: [ ] Passed & linked
4. Canary plan: [ ] Configured
5. Monitoring & alerts: [ ] In place
6. Rollback runbook: [ ] Available
7. Post-deploy validation: [ ] Ready
8. Approvals: Product lead _______ SRE _______ Security _______

Notes & evidence

Record links to CI runs, benchmark reports, runbook IDs, dashboards, and approval tickets here for the deployment record.