Machine vision pilot dataset & labeling checklist

Purpose

Use this checklist to plan and validate a focused, low-risk machine-vision pilot. The goal is to prove whether automated inspection can reliably detect targeted defects under real production variability and operator conditions. A well-scoped dataset, clear labeling rules, QA, and integration guidance reduce false positives, operator rejection, and wasted pilot effort.

Quick how to use

Walk the checklist with your pilot team (engineer, operator, quality lead, automation lead). For each item, record evidence, links or storage paths, and the person responsible. Prefer concrete examples, counts, and short runbook steps over vague statements.

Checklist

1. Define defect classes and OK examples

   Document each defect class you want the model to detect and show at least 5–10 clear, annotated OK examples for the same SKU/variant. Create a short label schema file describing class names, allowed ambiguities, and edge case rules.

   Evidence to record: label-schema link or file path; example images folder path; owner and date.

2. Capture representative images across production variability

   Collect images that reflect real variation: lighting conditions, conveyor speed, part orientation, tooling changeovers, upstream variability, shift differences, and both new and worn parts. Aim for an initial pilot dataset of at least 500–2,000 images distributed across classes, with minimum per-class targets (e.g., 100 images per defect class where feasible).

   Record: capture plan, sample counts per condition, storage path, and a short checklist confirming coverage of lighting, speed, and part variance.

3. Control fixturing & part presentation

   Note whether parts are presented consistently (fixtures, guide rails, gravity-fed) or vary in presentation. If presentation varies, capture that variance in data and consider adding a camera or fixturing changes for production deployment.

   Evidence: photos of camera setup, fixturing notes, tolerance for part misalignment.

4. Labeling rules, annotation tool, and QA sample

   Create a concise labeling guide that answers: what counts as a defect, how to annotate small/ambiguous defects, and how to treat overlapping defects. Choose an annotation tool and store a copy of the label schema in the dataset repository.

   Run a QA sample: randomly select 5–10% of labeled items (min 100) and perform independent review by a second labeler or QA engineer. Record inter-annotator agreement and common disagreements.

5. Dataset partitioning and edge-case holdout

   Partition into train / validation / test sets (typical splits: 70 / 15 / 15 by image or by production run to avoid leakage). Reserve an edge-case holdout set (rare lighting or failure modes) to validate model robustness.

   Evidence: counts for each partition, hashing method or run-based split, and link to partition manifest.

6. Acceptance metrics and targets

   Agree measurable targets before modeling. Example pilot targets (adjust to risk & cost): precision ≥ 90% for rejects (to avoid unnecessary scrappage), recall ≥ 80% for safety-critical defects, or acceptable false alarm rate per 1,000 parts. Define pass/fail rules for the pilot (e.g., production reject rate reduction, operator override rate under X%).

   Record: metric definitions, numeric targets, evaluation dataset to be used, and the person who will approve them.

7. Operator runbook for false positives / false negatives

   Prepare short operator instructions for handling model outputs: how to verify flagged parts, how to record overrides, and escalation steps for ambiguous cases. Include a simple log form (timestamp, image ID, model score, operator decision, reason).

8. Integration notes (trigger, reject, or assist)

   Decide how vision results will be used in the pilot: as an assistive prompt for inspection, trigger for automated reject, or a notification only. Document the mechanism (PLC I/O, MES flag, operator HMI) and validate signal paths with the control engineer.

   Evidence: diagram or short description, points of integration, required change control steps.

9. Monitoring, retraining plan, and versioning

   Define how you will monitor model performance in production (operator override rates, false alarms by shift, per-lot behavior). Set a versioning convention for dataset and model, and a retraining cadence or trigger (e.g., after N overrides or every 3 months).

10. Storage, metadata, and traceability

    Store images and annotations with clear filenames, metadata (timestamp, line, SKU, operator), and a manifest file. Record the annotation tool, annotator IDs, and labeling timestamps for traceability and audits.

Sample acceptance checklist (compact)

• Defect classes & OK examples: documented and stored — yes / no
• Representative images captured across shifts & lighting: yes / no
• Minimum per-class images collected: yes / no (counts documented)
• Labeling rules & QA sample passed: yes / no (agreement %)
• Training / validation / test partition created: yes / no
• Acceptance metrics defined and agreed: yes / no (values recorded)
• Operator runbook exists: yes / no
• Integration path defined: yes / no
• Dataset storage & versioning in place: yes / no (path)

How to close the pilot

Run evaluation on the held-out test set and measure the agreed metrics. Conduct a short production shadow test (model runs live, does not trigger rejects) for at least one shift or one lot. Compare operator override rates, false alarms, and missed defects. If acceptance metrics are met and operators report manageable workflow impact, prepare a documented plan for scale including PLC/MES integration, maintenance responsibilities, and retraining procedures.

Notes & links

Keep links to dataset storage, label schema, partition manifests, model evaluation reports, and operator runbooks in one accessible place (project folder or enterprise domain). Record owner and next review date.