AI Model Monitoring & Post‑Deployment Surveillance Runbook

Purpose and scope

This runbook helps healthcare teams keep clinical AI models safe, reliable, and equitable after deployment. It describes what to monitor, how often, who acts, and what to do when something goes wrong. Use this playbook for production clinical models that influence diagnosis, triage, treatment, or operational decisions. Adapt thresholds and cadence to model risk, regulatory requirements, and local workflows.

Key Hungers

• Detect model degradation, distribution shift, or safety signals quickly.
• Provide clear triage and incident response steps to protect patients.
• Maintain audit-ready logs, documentation, and communication templates.

Mal Hungers (Risks)

• Silent model failure, unnoticed bias, or drift leading to patient harm.
• Poor clinician trust due to unexplained or degraded model outputs.

Overview: monitoring strategy

Monitor three complementary domains continuously: operational health, predictive performance, and fairness/safety. Use automated daily health checks, weekly performance reviews, and quarterly audits.

Suggested cadence (adapt by risk)

• Daily: health & availability checks, input schema validation, data pipeline integrity.
• Weekly: performance and calibration review, output distribution and key metric trends.
• Quarterly: fairness audits, label quality checks, model retraining review, external validation if required.

Baseline performance checklist (daily / automated)

• Service availability: API latency and error rate within SLA.
• Input validation: missing fields, new categorical values, out-of-range numeric fields.
• Output sanity: probability ranges, NaNs, identical repeated outputs, confidence spike/dip.
• Data volume: expected throughput vs actual (sudden drops may indicate upstream problems).
• Logging: inference requests, model version, feature hash, downstream action flags stored.
• Alerting: health alerts route to on-call team with context (recent inputs, model version, request ID).

Weekly monitoring dashboard template

Include time-series charts and a short summary that can be reviewed in a 15–30 minute team huddle.

Essential charts and tables

• Primary performance metrics vs baseline (e.g., AUROC, AUPRC, sensitivity, specificity, PPV/NPV) with 90/95% confidence intervals.
• Calibration plot and calibration-in-the-small for key subgroups.
• Output distribution (prediction scores histogram) and changes vs previous week.
• Feature distributions for high‑importance features (detect covariate shift).
• Missingness and new category occurrence rates.
• Fairness metrics by protected attributes (e.g., demographic parity difference, equalized odds difference) where available and permitted.
• Alerts summary: open incidents, recent triage actions, and timestamps.

Suggested monitoring thresholds (example — tailor to context)

• Performance drop: >5% absolute decrease in primary metric (or >10% relative) triggers investigation.
• Calibration shift: predicted vs observed probability difference >0.05 in top decile.
• Output distribution drift: population-level KS statistic p<0.01 or a pre-agreed effect size trigger.
• Missingness increase: any key feature missing rate rise >50% relative to baseline.
• Fairness alarm: subgroup performance gap exceeds pre-accepted limit (e.g., sensitivity gap >0.10) or policy threshold.
• Latency: 95th percentile latency > SLA or error rate > 1% of requests.

Drift detection

Use statistical and model-aware drift detectors. Combine multiple signals to reduce false alarms.

Types of drift to detect

• Covariate drift: shift in input feature distributions.
• Prior/proportion drift: shift in base rate of outcomes.
• Concept drift: relationship between inputs and labels changed (hardest to detect without labels).

Methods

• Unsupervised detectors: population-level distance metrics (KL divergence, KS test), PSI, and embedding-space drift.
• Supervised detectors: monitor labeled outcome metrics when labels become available (delayed labels).
• Model-based change detectors: track feature importances or internal activations for structural shifts.

Alerting and triage workflow

Define alert levels and required actions. Keep the workflow simple and test it with regular drills.

Alert severity levels

1. Informational — anomaly observed but within tolerance. Log and monitor.
2. Warning — threshold exceeded (example: single-week metric drop). Owner investigates within one business day.
3. Critical — safety or large performance degradation (example: >10% drop). Immediate incident response and possible rollback.

Incident triage checklist

1. Record alert: timestamp, metric, model version, sample request IDs, recent deploys.
2. Assess scope: number of affected requests, clinical pathways touched, patient impact potential.
3. Replicate: run failing or anomalous inputs through a staging copy and baseline model.
4. Check provenance: recent data pipeline changes, labeling delays, feature engineering changes, code deploys, config toggles.
5. Escalate if necessary to clinical safety lead, legal/compliance, and risk management.
6. Decide temporary mitigation: disable automated actions, show advisory message to clinicians, or rollback to previous version.

Rollback and mitigation procedures

Rollback is a clinical safety action, not a technical convenience. Use clear criteria.

Rollback decision criteria (examples)

• Critical alert with evidence of patient‑facing harm or large metric drop meeting predefined thresholds.
• Undetermined root cause after initial triage and reasonable mitigations.
• Regulatory or compliance direction to pause model use pending investigation.

Rollback steps

1. Notify stakeholders (on-call ML engineer, clinical lead, product owner, compliance) and record the decision.
2. Switch traffic to prior validated model or turn off automated clinical recommendations; ensure graceful fallback messaging to users.
3. Lock write access to model configuration and rollforward to an incident branch for analysis.
4. Begin RCA (root cause analysis) and retain all relevant logs and artifacts.

Post-incident review and learning

• Complete formal RCA within the agreed SLA (e.g., 72 hours for critical incidents).
• Record findings, corrective actions, and preventive actions in the model change log.
• If retraining is required, verify training data quality, label drift, and revalidate on held-out test sets and clinically relevant subgroups.
• Schedule follow-up monitoring to validate recovery and prevention effectiveness.

Roles & responsibilities

• Model owner: accountable for monitoring plan, thresholds, and incident sign-off.
• ML engineer / operations: implement monitoring, runbooks, and execute rollbacks.
• Clinical safety lead: triage clinical impact and communicate with clinical teams.
• Data steward: ensure data quality, labeling processes, and data access governance.
• Compliance/legal: consult for patient communication and regulatory reporting.

Data & logging requirements

• Log every inference: timestamp, model version, input feature hashes, output score, decision/action taken, request ID.
• Store labeled outcomes when available and link to inferences for performance calculation.
• Preserve logs in immutable storage for audit (retention per policy).
• Protect PHI: apply encryption, role-based access, and data minimization consistent with HIPAA and local rules.

Fairness and safety audits

Run structured fairness audits at least quarterly or more frequently for high-risk models. Include subgroup performance, calibration by subgroup, and label quality checks. Document decisions and mitigations.

Communication templates

Clinician notification (short)

Subject: Model Alert — [ModelName] temporarily paused

Message: We have detected an issue affecting [ModelName] that may change its outputs. As a precaution, automated recommendations have been paused for [affected pathway]. Please follow standard clinical judgment. Contact the clinical safety lead at [contact]. We will provide updates within [X hours].

Patient-facing notification (example for material change)

We identified an issue with an AI tool used to support clinical decisions. Your care team has been notified and no automated decisions are being relied upon until we confirm normal behavior. If you have concerns about your care, please contact your provider.

Testing, drills, and continuous improvement

• Run tabletop incident drills twice per year covering at least one critical and one warning scenario.
• Simulate input distribution shifts in staging to validate detection and response speed.
• Maintain an evolving checklist and update thresholds based on operational experience.

Documentation & audit artifacts

Maintain for each model: monitoring plan, metric baselines, alert thresholds, incident logs, RCA reports, retraining records, fairness audit results, and communications. Make these artifacts available for internal audits and external regulators as required.

Suggested quick-start checklist (copyable)

1. Confirm instrumentation: inference logs include request ID, model version, inputs, outputs.
2. Implement daily health checks and route alerts to on-call channel.
3. Create weekly dashboard with listed charts and schedule a standing 15-minute review.
4. Define severity levels and ownership for alerts.
5. Practice rollback once in staging and run a tabletop drill.
6. Schedule quarterly fairness and label‑quality audits.

When to involve regulators / reportable events

Consult legal and compliance for incidents that materially affect patient safety, privacy breaches, or meet local reportable-event thresholds. Keep timeline of discovery, action taken, and communication for compliance review.

Appendix: practical examples & references

Include links to your internal data dictionaries, model change log templates, and live dashboard templates. Recommended external references: FDA AI/ML guidance, local regulator advisories, and industry best practices on model surveillance.

Image search phrase: ai model monitoring healthcare runbook