Model monitoring & operations runbook

Purpose & scope

This runbook helps operators, ML engineers, SREs, analysts, and product owners detect and respond to model degradation, drift, and operational failures after deployment. It focuses on monitoring signals, detection methods, alerting and escalation, immediate mitigations, investigation steps, and follow-up actions required to restore model value and maintain safety.

Quick reference (cheat sheet)

• Primary owners: ML Owner (product/feature), Model Steward (ML engineer), Infra/SRE, Incident Lead.
• Immediate mitigation options: switch to previous model version, enable safe fallback, apply deterministic guardrail, throttle/disable model outputs.
• Top monitors: performance vs baseline, input distribution drift, output plausibility, latency/infra health, fairness and safety signals.
• Escalation window: ack alert within 15 min, first mitigation within 60 min (or as SLAs require).

Roles & responsibilities

• Model Steward (ML Engineer): owns model health instrumentation, root-cause analysis, retraining, and rollout of fixes.
• Product / Feature Owner: decides business mitigation and customer communication priorities.
• SRE / Infra: monitors infra health, latency, availability; executes infra mitigations.
• Incident Lead: coordinates the response, communicates status, and ensures post-incident follow-up.

Core monitors (what to observe)

Each monitor should be implemented with a clear measurement, baseline/expected range, sampling cadence, and owner.

• Performance vs baseline: key business or proxy metrics (accuracy, error rate, AUC, precision/recall, conversion lift). Report both aggregated and slice-level metrics (by cohort, region, customer segment).
• Input distribution drift: population-level statistics on features (mean, variance, categorical counts), feature covariance changes, and proxy derived features. Use PSI, KL-divergence, or similar drift metrics per feature and combined signatures.
• Output plausibility: monitor output ranges, distribution changes, confidence/calibration shifts, and nonsensical predictions (e.g., impossible labels, out-of-range values).
• Latency & infra health: request latency P50/P95/P99, error rates, throughput, retries, queue lengths, and resource utilization.
• Downstream business KPIs: conversion, revenue, complaint rate, manual override rate—anything that reflects model impact on the business.
• Safety & fairness signals: flagged problematic outcomes, differential error rates across protected groups, complaint/appeal volumes.
• Data quality & completeness: missingness, schema changes, sampling ratio drift (training vs production), label delay/backfill behavior.

Detection methods & cadence

• Define baselines from a representative reference window (e.g., last stable rollout or test set) and compute daily/real-time comparison windows.
• Use both statistical alerts (PSI > threshold, metric deviation beyond N sigma) and business-rule alerts (drop in conversion > X%, sudden spike in manual overrides).
• Support both streaming (real-time) and batch (daily) monitors depending on use case. Real-time for high-risk or latency-sensitive models; daily/weekly for lower-risk analytics models.
• Implement anomaly detection for novel patterns, but pair with explainability probes to avoid noisy alerts.

Alerting & escalation (playbook)

Every alert should include: summary, metrics and evidence, time window, suggested immediate actions, and contact list.

1. Acknowledge — Alert must be acknowledged within 15 minutes by the on-call Model Steward or SRE.
2. Assess severity — Triage using impact criteria: customer-visible harm, safety/regulatory risk, revenue impact, security/privacy concerns, or degraded service level.
3. Immediate mitigation — Choose one or more of: rollback to previous model version, enable deterministic rule-based fallback, reduce model exposure (throttle traffic), return safe default, or disable model altogether for impacted endpoints.
4. Escalate — If severity is high or mitigation fails within SLA (e.g., 60 minutes), escalate to Incident Lead, Product Owner, and Compliance/Legal if required.

Suggested escalation matrix

• Severity 1 (customer harm, safety risk): notify Incident Lead, Product Owner, Exec on-call, Compliance.
• Severity 2 (significant degradation): notify Model Steward, SRE, Product Owner.
• Severity 3 (minor degradation or noisy alert): notify Model Steward to investigate during working hours.

Immediate mitigation checklist

• Confirm alert validity (verify metrics with raw samples).
• If confirmed and high impact: rollback to last known-good model version or enable rule-based guardrail.
• Apply temporary throttling or reduced exposure to a subset of traffic (canary rollback by region/customer segment).
• Notify downstream teams and, if user-facing, prepare customer messaging templates if required.

Investigation steps (post-mitigation)

1. Collect evidence: recent inputs (sampled), outputs, model version, inference logs, feature pipelines, data schema changes, deployment events, code commits, infra changes, and external events (market, seasonality).
2. Reproduce the issue on a staging or replay environment using captured requests when possible.
3. Check training/validation drift: has the production data distribution diverged from training or validation windows?
4. Review upstream data pipelines for schema or preprocessing changes, missing features, or label delay/inconsistencies.
5. Run targeted unit tests: feature sanity checks, model scoring unit tests, and end-to-end tests with golden inputs.
6. Assess whether fixes require retraining with fresh data, model architecture change, feature engineering adjustments, calibration, or non-model (business) fixes.

Post-incident actions (RCA & remediation)

• Complete a brief incident report within 48 hours summarizing cause, mitigation, timeline, and impact.
• Define remediation plan: retrain schedule, data-collection fixes, enhanced monitoring, test coverage, and deployment policy changes.
• Schedule follow-up validation and a safe rollout plan (canary + progressive rollout with tighter metrics).
• Update runbooks, dashboards, thresholds, and owner contact lists based on lessons learned.

Operational tasks & cadence

• Daily: health check dashboard review, automated alert triage for critical monitors.
• Weekly: slice-level performance review, drift reports, and manual spot checks of flagged samples.
• Monthly: data-label quality review, retraining candidate assessment, and capacity planning with SRE.
• Quarterly: governance review including fairness audits, privacy compliance checks, and model inventory reconciliation.

Instrumentation & data requirements

To make monitoring effective, ensure the following are logged and accessible:

• Request/response pairs (inputs and outputs), with timestamps and model version IDs.
• Feature extraction traces and preprocessing provenance.
• Latency and error traces with stack or service identifiers.
• Downstream labels or delayed feedback when available, with label timestamps.
• Metadata: customer segment, request context, and sampling flags for privacy-sensitive fields.

Dashboards & reports

• Operational dashboard: latency, error rate, traffic volume, model version usage.
• Health dashboard: key performance metrics vs baseline, feature drift heatmap, output plausibility counter.
• Business impact dashboard: downstream KPIs (conversion, revenue, complaints) with model attribution when feasible.
• Alert inbox: active alerts, recent incidents, and open action items.

Testing & release controls

• Require canary deployments with monitored metrics and automatic rollback triggers for measurable regressions.
• Maintain a last-known-good model version and an automated rollback path.
• Use replay tests and shadow traffic to validate new models before routing live decisions.

Governance, compliance & safety

Escalate to compliance/legal for alerts with potential regulatory, privacy, security, or safety implications. Keep an up-to-date model inventory and justification for monitoring and retention policies. Retain sampled logs in accordance with privacy rules and data retention policy.

Common failure modes & diagnostic hints

• Sudden feature distribution shift — check upstream ingestion/ETL and third-party data sources.
• Gradual performance decay — likely need retraining with fresh labeled data or addressing label drift.
• Spikes in low-confidence outputs — check data quality or calibration; consider confidence-based throttling.
• Latency or error spikes — investigate infra, model size, batching behavior, or recent deployments.

When to call a human-in-the-loop

Trigger human review when outputs affect safety, legal/regulatory outcomes, high-value transactions, or when confidence/calibration crosses a critical threshold. Define concrete criteria per model and include these in the model's onboarding checklist.

Templates & artifacts to keep with the runbook

• Alert template with required fields (metric, evidence, time window, suggested actions).
• Incident report template (timeline, scope, impact, RCA, remediation plan).
• Rollback procedure and playbook for the deployment platform.
• Checklist for on-call handover and post-incident review agenda.

Where to improve next (opportunities)

• Structured incident logging form to standardize captures and enable analytics.
• Automated drift remediation pipelines (data collection, candidate retraining pipelines, and gated redeployments).
• Integrate monitoring with model registry and CI/CD to link incidents to commits and model artifacts.

Keep this runbook adjacent to the model's onboarding checklist and model card so each deployed model has tailored thresholds, owners, and human escalation contacts.