Predictive Maintenance Pilot: Data Collection Template & Evaluation Dashboard

Purpose

This template helps teams scope and run a small, focused predictive maintenance (PDM) pilot that proves whether sensor signals, data quality, and maintenance processes can produce actionable alerts that reduce unplanned downtime. Use it to capture pilot decisions, standardize labeling, and evaluate model usefulness with operational metrics — not only machine-learning metrics.

Pilot at-a-glance

• Pilot size: 3–10 similar assets or components (same model, failure modes, and operating context).
• Baseline collection: Collect raw signals and maintenance history for a baseline period (typically 4–8 weeks or longer for rare failures).
• Pilot run: 8–16 weeks of live alerts and monitored outcomes (adjust based on failure frequency).
• Primary owner: Reliability engineer or maintenance manager.
• Operational owner for alerts: Assigned technician or shift lead with clear action playbook.

Asset description

For each asset include:

• Asset ID, manufacturer, model, serial number
• Location (line, cell, machine bay)
• Operating profile (hours per day, typical loads, shift patterns)
• Relevant maintenance history (recent repairs, part replacements, preventive schedule)

Failure modes & labeling rules

Define the specific failure modes the pilot targets (e.g., bearing fatigue, motor winding insulation breakdown, belt fracture). For each mode specify how failures will be labeled:

• Failure event label fields: asset_id, failure_type, failure_start_timestamp, failure_end_timestamp (if applicable), root_cause (if diagnosed), corrective_action, ticket_id (CMMS), severity.
• Labeling rules: label a failure when corrective action or part replacement occurred because of abnormal condition. Do not label routine scheduled maintenance as failure unless caused by the target failure mode.
• Near-miss / precursor labels: optionally record close-calls where abnormal condition was corrected before a failure—useful for improving model recall.
• Annotation examples: “2026-06-10T03:15Z — bearing inner race replaced after vibration spike and acoustic emission; root_cause=inner_race_crack; ticket=1234.”

Signals required & sampling guidance

Choose signals tied to the chosen failure modes. Typical signal categories and guidance:

• Vibration (accelerometer): frequency-domain information is valuable for bearing and rotating faults. Typical sampling depends on rotational speed and fault frequencies — common pilots use 1 kHz to 25 kHz for detailed analysis; if unsure, start with lower rates (1–5 kHz) and validate in an exploratory stage.
• Velocity / RPM / tachometer: sample at sufficient resolution to align vibration to revolutions (timestamped).
• Temperature: 1 sample per 10s to 1 sample per minute is a common starting point for thermal trends.
• Current / power / torque: 1–10 Hz typical for electrical signatures of load changes.
• Discrete signals / status bits: event-driven (timestamped) whenever state changes.

Record wall-clock timestamps in UTC, ensure synchronized clocks, and include sensor health telemetry (battery, signal-to-noise ratio, self-test flags).

Data quality checks

Run these checks before trusting model outputs:

1. Missing data rate per sensor (target: <5% in pilot windows; acceptable depends on context).
2. Out-of-range values and sensor saturation events.
3. Timestamp skew between sensors (align within one sampling period or account for offset in preprocessing).
4. Noise floor and SNR estimates — ensure signal contains relevant information for the failure mode.
5. Label completeness and label latency (how long after an event is the label available?)
6. Versioning policy for data processing and feature extraction pipelines.

Labeling guidance & dataset structure

Store a tidy table of labeled events and a linked time-series table for signals. Minimal columns:

• Events table: event_id, asset_id, failure_type, start_ts, end_ts, ticket_id, corrective_action, diagnosed_by, confidence
• Time-series table: ts, asset_id, sensor_id, sensor_type, value, sample_rate, quality_flag

Keep a labels-change log to track corrections and audits.

Pilot duration & sample-size guidance

• Choose duration based on expected failure interval. If failures are weekly, 8–12 weeks may be sufficient; if monthly or rarer, extend pilot or use simulated/retrospective labeled data.
• Ensure at least 10–30 labeled failure events across pilot assets for initial model training/validation where possible; otherwise treat results as exploratory and focus on signal validation.

Success criteria (define before running)

Operational success should be expressed in both ML and business terms. Example criteria:

• Actionability: ≥ 60–70% of alerts lead to a documented, intended maintenance action within the defined lead time.
• Precision / False-alarm rate: Precision ≥ 0.6 (adjust target to business tolerance). Track false-alarm cost per month.
• Lead time: Median lead time from alert to failure >= minimum time required to schedule corrective maintenance without production disruption.
• Downtime impact: Measurable reduction in unplanned downtime for pilot assets compared with baseline period (example target: 10% relative reduction — set realistic, evidence-based targets).
• Integration readiness: Alerts are integrated into the CMMS or operator workflow with clear owners and playbooks.

Declare pass/fail thresholds before starting and include an “inconclusive” bucket if sample size is too small.

Operational playbook for alerts

1. Alert received by assigned owner (who and how).
2. Immediate check steps (visual/audio/temp quick check).
3. Decision tree: defer to scheduled maintenance vs. urgent intervention.
4. Create CMMS ticket & record action taken and result (true positive, false positive, near-miss).
5. Label the outcome in the events table.

Dashboard widget specification (what to show and why)

• Alerts over time: count of alerts by asset and alert severity; helps spot alert storms and sensor issues.
• Confusion matrix (time-windowed): True Positive / False Positive / False Negative / True Negative computed against labeled failure windows. Define prediction window (for example: an alert counts as TP if failure occurs within X hours/days after alert).
• Precision / Recall / F1 over time: show trends and confidence intervals.
• Time-to-failure distribution for true positives: visualize lead times to ensure alerts provide actionable windows.
• Maintenance actions taken: list actions, CMMS tickets, and resulting downtime durations.
• Data quality panel: missing data %, sensor health events, timestamp drift alerts.

Each widget should include a short interpretation note: what does improvement look like and what team action should follow.

How to use this template (step-by-step)

1. Assemble pilot team (reliability owner, data engineer, plant supervisor, CMMS lead, operator representative).
2. Pick 3–10 assets and define failure modes and success criteria in this template.
3. Install sensors, verify signals with initial data-quality checks, and collect baseline data.
4. Label historical failure events if available; run exploratory analysis to confirm signal relevance.
5. Train a simple model or rule-based detector; deploy to generate alerts in parallel with normal operation (no automatic shutdowns during pilot).
6. Follow the operational playbook for each alert; record outcomes and label them promptly.
7. Review dashboard weekly; evaluate against pre-defined success criteria at pilot end.
8. Decide: scale, iterate (improve sensors/labels/playbooks), or stop and document learnings.

Common failure modes to avoid

• Poorly selected assets (don’t mix unrelated failure modes in the same pilot).
• Insufficient labeled events (limits statistical conclusions).
• No integration with operator workflow (alerts ignored).
• Not versioning datasets and models (cannot reproduce results).

Appendix — Example: Bearing wear pilot (short)

Assets: conveyor motor bearings (n=6). Signals: accelerometer (4 kHz), tachometer, temperature (1/min). Labels: bearing_replace events from CMMS. Success criteria: precision ≥ 0.65, median lead time ≥ 72 hours, reduction in unplanned downtime ≥ 15% vs baseline. Pilot duration: 12 weeks.

Use

Copy this template into your pilot record, fill all fields, and attach baseline data samples and labeling examples. Use the dashboard spec to build a minimal evaluation view that your team reviews weekly. Treat unclear results as signals to iterate on labeling and signal selection rather than reasons to expand prematurely.