Anomaly Detection Tuning & Operator Runbook

Welcome — turn noisy signals into trusted actions

When anomaly alerts are precise, operators respond quickly and incidents are stopped before they cascade. When alerts are noisy or vague, people ignore them and real problems slip through. This playbook helps you define what matters, tune alerts for low false‑positive noise, create operator runbooks that are short and testable, and build a feedback loop so models and rules get better with real outcomes.

What this playbook contains

• Anomaly definition checklist to make detections meaningful and actionable
• Tuning workflow and a practical tuning worksheet you can use on historic data
• Alert tiering and escalation tree for consistent responses
• Sample operator runbooks for common shop‑floor anomalies
• Metrics and a feedback loop for continuous improvement

Anomaly definition checklist

Before you tune thresholds, agree what the anomaly actually means. Use this checklist to make definitions precise and testable.

1. Business impact: Which measurable harm are we preventing (safety, scrap %, downtime minutes, quality escape, customer complaint)?
2. Observable signal(s): Which sensor(s), quality metric, or derived feature will indicate the anomaly?
3. Expected pattern: Define pattern (single value spike, sustained drift for N minutes, unusual combination of values, missing heartbeat).
4. Operational context: Which lines, shifts, or SKUs does this apply to? Are there scheduled reasons for the pattern (start‑up, changeover)?
5. Actionability: What immediate operator action(s) can reasonably reduce the impact within the SLA?
6. Escalation trigger: When does it escalate (time, repeat occurrences, severity)?
7. Success definition: How will we know the alert produced the desired result (reduced downtime, contained scrap, verified root cause)?

Tuning workflow — practical steps

Use a small, repeatable process so tuning is measurable and reversible.

1. Collect baseline: Pull a historical window containing recent incidents and normal operations (include changeovers and unusual but valid events).
2. Label incidents: Mark true incidents, false alarms, and non‑events in the historical set.
3. Simulate thresholds: Run candidate thresholds or model scores against the labeled set to estimate precision/recall and false positive rate.
4. Choose tiering: Map scores to tiers (Info / Action / Critical) with different response SLAs.
5. Pilot with owners: Deploy on one line or shift with a human-in-the-loop and collect feedback for 1–4 weeks.
   • Record every alert outcome using the runbook outcome form.
6. Review and adjust: Measure metrics (below), adjust thresholds, and repeat until acceptable tradeoff between missed incidents and noise is reached.
7. Promote to production: After pilot success, roll out with documented ownership and periodic review cadence.

Tuning worksheet (fields to capture)

• Detection name / ID
• Signals & windows used
• Candidate thresholds or model score boundaries
• Label counts: true positives, false positives, false negatives
• Precision, recall, false positive rate
• Pilot dates and scope
• Owner (person/team) and review cadence

Alert tiering & escalation tree

Not all anomalies require the same response. Use tiers so operators prioritize correctly.

1. Informational / Monitor (low priority)
   • When: low score, single deviation, no immediate business impact
   • Who sees it: line engineer dashboard, shift log
   • Action: monitor and document; no immediate intervention
2. Action required (medium priority)
   • When: sustained drift or repeat events, potential scrap or slowdown
   • Who: operator + supervisor
   • Action: follow runbook; if unresolved in X minutes escalate to maintenance
3. Critical / Safety & Stop (high priority)
   • When: immediate safety or severe production risk
   • Who: operator, supervisor, maintenance, safety officer
   • Action: stop line if required, follow emergency runbook, notify leadership

Sample operator runbooks (short, testable)

Each runbook should fit on one page and follow this compact template: Situation → Immediate steps → Verification → Escalation conditions → Log outcome.

Runbook A — Vibration spike on motor X

1. Situation: Vibration RMS exceeded 3.5 g for 30 seconds on motor X (Action tier).
2. Immediate steps:
   1. Check machine speed and loading. Reduce load to idle if safe.
   2. Visually inspect bearings, coupling, and mounting fasteners for obvious looseness or damage.
   3. Record vibration waveform and tag sample for maintenance.
3. Verification: Vibration returns to baseline within 10 minutes and production rate normalizes.
4. Escalate to maintenance if vibration remains > threshold after 10 minutes or if unusual noise/heat is present.
5. Log outcome: actions taken, time to resolution, suspected cause, whether corrective maintenance was scheduled.

Runbook B — Product dimension drift (quality KPI spike)

1. Situation: 3 of 5 sampled pieces exceed spec on dimension Y during last sampling cycle.
2. Immediate steps:
   1. Stop line if out-of-spec risk is high; otherwise slow down and increase sampling frequency to every 5 minutes.
   2. Check latest setup parameters against last known-good setup. Reapply setup if discrepancy found.
   3. Notify quality and record a hold on suspect batch.
3. Verification: next 10 samples within spec and root cause identified (tool wear, setup drift, material defect).
4. Escalate to process engineer if unable to stabilize within 20 minutes.
5. Log outcome and attach sample measurements.

Alert text & context template

Good alerts are short, specific, and include context and immediate action. Use this template:

[TIER] • [Detection name] • [Line/Asset] • [Time] • Key evidence: [metric=value, window] • Action: [one‑line instruction] • Escalate if: [condition and who]

Metrics — how to measure alert quality

• Precision (positive predictive value): true positive alerts / total alerts — reduces noise.
• Recall (sensitivity): true positive alerts / total real incidents — avoids missed incidents.
• False positive rate: false alarms per 1,000 operating minutes or per shift.
• MTTA (mean time to acknowledge): from alert to operator acknowledgement.
• MTTR (mean time to resolve): from alert to verified resolution.
• Runbook execution rate: percent of alerts where runbook steps were followed and logged.

Feedback loop & continuous improvement

1. Record every alert outcome in a simple incident form (who, what, actions, resolution, labels: TP/FP/FN).
2. Weekly triage: owners review recent alerts, relabel questionable cases, and recommend threshold changes.
3. Monthly model/rule review: retrain models using newly labeled incidents and test candidate thresholds on holdout data.
4. Change control: record threshold or model updates, owners, and rationale.
5. Periodic drills: test critical runbooks quarterly to ensure operators know steps under pressure.

Governance & ownership

• Owner: who is responsible for each detection (operator team, process engineer, data scientist)?
• Review cadence: pilot → weekly triage → monthly model review → quarterly drill.
• Documentation: store runbooks, tuning worksheets, and audit trail in a shared collection so local plants can copy and adapt.

Quick checklist for a pilot deployment

• Pick one detection and one pilot line
• Run simulation against 30–90 days of historical data
• Define tiers and initial thresholds
• Create a one‑page runbook and a short alert message template
• Collect alert outcomes using a simple form and meet weekly for 4 weeks
• Adjust, document, and expand rollout

Where to start now

If you have one hour: identify the top two alerts that create the most noise or the highest business impact. Apply the anomaly definition checklist to both, pick conservative thresholds, and write a one‑page runbook. Run a quick simulation on recent data and schedule a 1‑week pilot.

Notes: This playbook is designed to be copied and tailored. Local teams should adapt thresholds, context, and escalation rules to their equipment, safety requirements, and operating rhythms.