Model deployment & operational guardrails for OT

Why this guide exists

Deploying ML to operational technology (OT) systems creates real value — and real risk. This guide helps teams run short, shop‑floor experiments that prove safe, observable, auditable model lifecycle practices. It focuses on practical acceptance checks, monitoring signals you can implement quickly, clear rollback and operator‑override rules, and the minimum governance artifacts you must capture so experiments can scale without introducing safety, reliability, or compliance gaps.

Intended audience

ML engineers, OT engineers, plant managers, operators, safety and quality leads, and anyone responsible for piloting or operating ML models on the shop floor.

Quick start: mandatory annex for any OT model pilot

Use this guide as a required annex to any model deployment to the shop floor. Before any model can act on control decisions or present suggestions to operators, the pre‑deployment checklist and the runbook below must be completed and approved by the designated stakeholders.

Pre-deployment checklist (minimum)

Each item should be documented and attached to the pilot package.

1. Test dataset and evaluation
   • Hold‑out test set drawn from operational data, including edge cases and known failure modes.
   • Performance metrics with acceptance thresholds (example: FPR < 2%, recall > 90% for safety alarms; MAPE < 10% for regression predictions). Use domain‑appropriate measures.
   • Confusion matrix, ROC/AUC or precision‑recall curve as relevant. Report per‑shift and per‑machine breakdowns where possible.
2. Data readiness and completeness
   • Verify sensor health, sampling rates, missing data patterns, and timestamp alignment.
   • Document feature provenance, transformations, and any enrichment steps.
3. Safety & control envelope checks
   • Define safe ranges for model outputs. Any output outside the envelope must be treated as invalid and trigger operator review or automatic safe state.
   • Simulate model outputs in a safe environment (digital twin or replay) across normal and fault scenarios.
4. Human‑in‑the‑loop rules
   • Who may accept or override model recommendations? Document exact override workflows and UI/confirmation steps.
   • Define when model outputs are advisory only versus when they may trigger automated actions.
5. Acceptance & approvals
   • Sign‑offs needed before deployment: ML owner, OT/controls owner, safety officer, operations lead, and the pilot owner.
   • Attach an explicit rollback approval and communications plan.

Runbook for monitoring (operational signals and implementation)

Monitor a combination of model health, input data health, system performance, and business/operational KPIs. Implement tiered alerts (info, warning, critical) and who they notify.

Minimum signals to monitor

• Latency: request/response time. Threshold example: warning at 2× normal latency, critical at 5× or any latency that breaches control loop timing constraints.
• Input completeness: percent of expected features present. Alert when < 95% on a sustained window (e.g., 5 minutes).
• Input distribution / drift: track population stability index (PSI), KL divergence, or feature z‑score shifts. Example trigger: PSI > 0.2 sustained for 1 hour → warning; PSI > 0.4 → critical.
• Prediction distribution: sudden shifts in predicted class proportions or confidence scores. Example: predicted failure probability increases by factor X or median confidence drops below threshold.
• Model performance proxies: where ground truth arrives with delay, use proxy signals such as follow‑up sensor patterns or operator feedback. Define acceptable decay rates (for example, predicted quality defect rate +5% vs baseline over 24 hours is a warning).
• Business/operational KPIs: OEE, throughput, defect rate. Set realistic guardrails so models that harm KPIs are quickly identified.

Monitoring cadence & dashboards

Log metrics at appropriate granularity (per batch, per shift, or per second depending on process). Provide a lightweight dashboard for the operations team and an alerting channel (SMS/email/ops chat) for critical events.

Escalation & rollback triggers

Define deterministic, testable triggers that cause one of three actions: operator alert only, degrade to advisory, or automatic rollback to known good model. Keep rollback fast and well‑tested.

Example automatic rollback triggers

• Critical safety envelope breach in model output or actuator command.
• Instrumented data shows missing critical inputs for > N seconds/minutes.
• Model confidence drops below threshold combined with surrogate performance deterioration (e.g., KPI drop > X%).
• Unrecoverable latency that violates control loop timing.

Rollback runbook (step by step)

1. Automated system flags event and attempts safe state (if applicable).
2. Notify operator and on‑call ML/OT engineer with event packet (timestamp, input snapshot, model version, recent metrics).
3. System switches to previous stable model (canary rollback) and marks the problematic model as quarantined in registry.
4. Incident owner opens a ticket and begins diagnosis with required logs attached. Notify production leadership if downtime or safety risk occurred.

Versioning and retraining cadence guidance

Maintain a model registry mapping code, data snapshot, training pipeline, hyperparameters, and approval artifacts. Use semantic versioning (major.minor.patch) and record readable change notes.

• Retrain triggers: sustained performance degradation beyond threshold (example: validation metric drop > 5% vs baseline), or significant input distribution drift (PSI > 0.2).
• Canary / incremental rollout: start at a small traffic slice (e.g., 5–10%), monitor for N hours/shifts, then increase to 50% and finally full rollout if stable.
• Schedule periodic retrain windows when appropriate — for many OT problems, a monthly cadence is a starting point, but use data to shorten or lengthen intervals.

Logging, audit trail and governance artifacts

Capture immutable records for traceability and audits:

• Input snapshot (anonymize sensitive fields) for each decision or a sampled subset.
• Model version, configuration, and container/digest used.
• Decision outcome, confidence, and any automated action taken.
• Operator overrides and rationale with timestamp and operator id.
• Alerts, escalation steps taken, and rollback events.
• Link to training dataset snapshot, preprocessing code, and evaluation artifacts.

Retention: align with site policies and regulatory requirements. For pilots, preserve full trails for the duration of the pilot plus a minimum archival window (example: 1 year) so lessons can be reviewed.

Roles & responsibilities (minimum)

• ML Owner: model performance, retraining triggers, and evaluation artifacts.
• OT/Controls Owner: safe operating envelope, integration, and final approval for automated actions.
• Operator: approve/override model recommendations; execute overrides per documented workflow.
• Safety/QA: safety case review and sign‑off for any model that can affect process safety or product quality.
• On‑call ML/OT engineer: incident response and investigation after alerts or rollbacks.

Templates & example thresholds

Attach a simple acceptance table to every pilot. Example:

Common mistakes to avoid

• Deploying models without provenance for the training data or transformation code.
• Lack of clear operator override paths or assuming operators will ‘figure it out’.
• No plan for slow‑burn drift — waiting months to check model performance.
• Not preserving input snapshots and operator rationale for post‑incident analysis.

How to use this guide in your site

1. Adopt as mandatory annex to any model deployment that touches OT. Require the pre‑deployment checklist sign‑off.
2. Run a short pilot using canary deployment and the monitoring runbook. Keep the pilot scope small (one line/machine/shift) and iterate fast.
3. Collect logs and incident artifacts via your existing central logging. Post‑pilot, run a review, capture lessons, and refine thresholds and procedures.
4. When patterns stabilize, codify the approved runbook into site standards and add the model package to the model registry with clear ownership and retraining cadences.

Next practical steps

• Create a minimal dashboard for the signals listed in this guide and set one critical alert to route to the on‑call engineer.
• Run a 2‑week canary on noncritical equipment to validate monitoring and rollback procedures.
• After the pilot, produce a short ‘playbook’ one‑pager for operators describing what to expect and how to override safely.

Preserve this guide with each model package and treat it as a living document: update thresholds and procedures based on pilot results, regulatory changes, and evolving operational realities.