Anomaly Detection Response Runbook

Purpose

This runbook translates anomaly detections into clear, safe, repeatable operator actions and investigation steps so teams reduce false alarms, catch meaningful problems earlier, and close the loop on incidents. Use this as the primary operator reference for handling any shop‑floor anomaly alert produced by rules or models.

High-level Triage Flow (one‑page)

1. Acknowledge: Confirm receipt of the alert in the system within the target SLA (e.g., 5 minutes).
2. Safety Check: Immediately verify whether the anomaly creates any safety, fire, or environmental hazard. If yes, follow emergency procedures and notify Safety immediately.
3. Containment: If the anomaly could cause product loss, scrap, or further escalation, pause the affected line, isolate the batch, or take the minimum intervention needed to stop harm.
4. Capture Data: Collect the standard alert dataset (see Data to Capture section) before changing anything that would erase evidence.
5. Quick Validate: Run the Quick Validation Checklist to determine whether the alert is likely meaningful, a transient sensor glitch, or a known nuisance.
6. Decide: If valid and actionable -> follow corrective action / escalate. If likely false -> mark with reason and close. If uncertain -> escalate to SME with captured data.
7. Post‑incident Review: Complete the Post‑Incident Review Form within 24–72 hours and file the record for metrics and tuning decisions.

Immediate Safety & Containment Steps (What to do first)

• If there is any risk to people, stop work and follow the facility Emergency Response Plan. Call Safety and local first responders as required.
• If the anomaly threatens product quality or process stability, switch the line to safe mode (reduced speed, hold feed) or stop the machine per local standard work.
• Physically secure affected product (tag, segregate, quarantine) with clear labels and timestamps.
• Do not reset sensors, clear buffers, or restart systems until required evidence is captured unless doing so is necessary for safety.

Data to Capture for Every Alert

Collect the following minimum dataset for each anomaly alert. Saving this data consistently enables reliable post‑incident review and model/rule tuning.

• Alert ID, Timestamp of detection, and Timestamp acknowledged by operator
• Alert source: model name / rule name / sensor ID
• Raw signal values and model score/confidence at detection (include the threshold used)
• Machine/line ID, part/lot/batch ID, shift, operator on duty
• Operating context: line speed, temperatures, recent maintenance, recipe/recipe step
• Operator observations (text) and corrective actions taken
• Photographs, screenshots, or short video clips linked or attached
• Related alarms or events in the preceding 30 minutes
• Final disposition: false positive / true positive / unknown / known issue

Quick Validation Checklist (use at time of alert)

Answer these yes/no questions. If two or more answers are "no" or if any safety question is "yes," escalate immediately.

1. Is the equipment operating within normal setpoints? (yes/no)
2. Are sensors and cables visibly intact and connected? (yes/no)
3. Is the anomaly correlated with a process change (e.g., tool change, recipe switch)? (yes/no)
4. Do other sensors on the same subsystem show abnormal readings? (yes/no)
5. Is there a recent maintenance activity that could explain the signal? (yes/no)
6. Is there visible product or process evidence (e.g., defect, leak)? (yes/no)

Interpretation: mostly "yes" and visible evidence -> likely meaningful. Mostly "no" with single-sensor spike -> likely transient or sensor fault; document and monitor.

Escalation Rules (who to call, when)

Use the minimum escalation needed to resolve the situation safely and promptly.

• Immediate Safety Risk: Call Safety Lead and Plant Manager now. Do not wait.
• Product Loss or Line Halt: Notify Shift Lead and Maintenance within target SLA (e.g., 15 minutes).
• Quality Impact (potential customer escape): Notify Quality Engineer and Production Manager within 30 minutes.
• Unknown Technical Cause after quick validate: Escalate to SME (Process Engineer or Data Scientist) with captured data within 60 minutes.
• Recurring Alert Pattern (>3 in 24h): Open an incident ticket and schedule a root cause review with Maintenance, Quality, and Process.

Maintain a simple escalation matrix in local standard work with names, roles, and contact method (phone, pager, team chat channel).

Corrective Action Templates (examples)

• Sensor glitch suspected: Record data, tag sensor, perform sensor swap or inspection per LOTO and sensor replacement standard work. Mark alert as 'suspected sensor' and monitor.
• Process deviation: Revert to last known good recipe/parameter, hold product, trace back affected lots, and run sample checks to determine scope.
• Equipment fault: Isolate equipment, notify Maintenance for diagnostic, capture logs, and follow maintenance troubleshooting checklist.

Post‑Incident Review Form (template)

Complete this form within 24–72 hours. Store it with the alert record.

1. Alert ID & Date
2. Summary of what happened (200 words)
3. Data captured (reference files/links)
4. Root cause (confirmed / suspected / unknown)
5. Corrective actions taken
6. Permanent fixes planned (yes/no) — description and owner
7. Was the alert actionable and useful? (yes/no) — why or why not
8. Suggested tuning or rule changes (include proposed new threshold or feature)
9. Metrics impacted (e.g., scrap, downtime minutes, customer impact)
10. Reviewers and approvals (names, roles, date)

Metrics to Measure Alert Quality and Response

• False Positive Rate (FPR): FP / (FP + TP) measured weekly. Track by alert source and line.
• Precision (per alert source): TP / (TP + FP). Set target precision based on business risk (e.g., 0.8 for production-critical alerts).
• Mean Time to Acknowledge (MTTA): median time from detection to operator acknowledgment.
• Mean Time to Resolve (MTTR): median time from detection to closure or containment.
• Percent of alerts with complete data: percentage of alerts that include required attachments and fields.
• Escalation Accuracy: percent of escalations that required SME intervention (helps detect over-escalation).

Owner: assign a single Alert Quality Owner (often Operations or Process Engineering) to review metrics weekly and run tuning experiments monthly.

Tuning Guidance (rules vs models)

Use this decision rule when choosing whether to tune a threshold, change a rule, or retrain a model:

• If false positives are concentrated on a single, stable pattern (e.g., specific product line conditions): prefer targeted rule exceptions or pre‑processing filters.
• If many meaningful events are missed (false negatives): investigate model recall and consider retraining with representative examples.
• If alerts are noisy and random: inspect sensor health, add data quality checks, and avoid lowering thresholds further—this increases false positives.
• When in doubt: run a controlled A/B test where one threshold runs in monitor mode while the other actively triggers actions; compare precision, recall, and operational burden.

How to Improve Operator Trust

Operator trust is essential. Consider these practical practices:

• Provide short context in the alert (why it flagged: feature values, recent change).
• Allow operators to mark alerts as "verified false" quickly and capture a short reason (helps training data).
• Keep the validation checklist short and integrated into the alert UI or paper board for shift use.
• Share weekly summaries highlighting when alerts prevented bigger problems (builds positive narrative).

Local Adaptation & Versioning

Treat this runbook as a living document. Sites should keep a local copy that includes their escalation contacts, phone numbers, and any plant‑specific containment steps. Record version, last review date, and owner on the document.

Appendix — Example fields for system entry

These are structured fields that the alert record should capture (useful when converting this runbook into an interactive form):

• alert_id, detected_at, acknowledged_at, closed_at
• alert_source, model_score, threshold_at_detection
• machine_id, line_id, part_id, batch_id
• operator_notes (textarea), attachments (photos/video links)
• quick_check_answers (structured booleans), disposition (enum: true_positive / false_positive / unknown / known_issue)
• post_incident_review (link to review form record)

Next steps for teams

1. Adopt this runbook as local standard work and fill the escalation matrix with local contacts.
2. Start capturing the full dataset for each alert (even if initially manual) for 30 days.
3. Review alert metrics weekly and run focused tuning experiments monthly.
4. Consider converting the Post‑Incident Review and alert data capture into an interactive form to standardize records and enable analytics (see capability notes).