Machine Vision Pilot: Dataset Collection & Labeling Guide

Purpose

This guide helps teams run a focused, low‑risk machine vision pilot by collecting and labeling a dataset that can produce a deployable model. It keeps the pilot practical: define a narrow target, gather representative images, label consistently, check labeling quality, and verify model performance against meaningful acceptance criteria before committing to scale.

Pilot scope and success criteria (start here)

Before you collect data, agree on these items with the team (engineering, operators, quality, and IT):

• Target defects or detections: clearly describe each class you want the model to detect (e.g., small scratch on the bearing face, missing O‑ring, incorrect label).
• Operational acceptance criteria: desired precision/recall or F1 thresholds, acceptable false positive/false negative costs, inference time target on the intended hardware, and required detection range/magnification.
• Deployment constraints: target camera resolution, compute platform (edge CPU/GPU), lighting limitations, and whether parts will be fixtured or free‑flowing.
• Decision rule for scaling: e.g., model meets recall ≥ 0.90 and precision ≥ 0.85 on test set and passes an operator A/B validation for 1 shift.

Designing the dataset

Good datasets reflect the real variation your model will see. Plan for:

• Representative diversity: include parts from multiple production lots, shifts, machines, operators, and relevant process states (wet/dry, different lubricant, finished/unfinished).
• Environmental variation: controlled and uncontrolled lighting, reflections, background clutter, and camera angle/height differences you expect in the pilot.
• Class balance planning: defects are often rare. Collect sufficient negatives and consider oversampling or targeted collection of defect cases (see sizing guidance below).
• Negative and near‑miss examples: collect examples that are close to but not defects—these reduce false positives.

Camera placement, fixturing, and part presentation

Small changes here dramatically affect model success.

• Fix the camera to reduce variable framing. If parts must move, standardize conveyor position and speed.
• Keep consistent distance, angle, and field of view for the feature of interest; document exact coordinates and lens/focal settings.
• Use simple fixtures to reduce background variability when practical. If you must inspect free‑flowing parts, collect representative motion blur and orientations.
• Annotate whether images are fixtured or free‑flowing in metadata.

Lighting and imaging best practices

• Prefer diffuse, stable lighting. Avoid specular highlights by using polarizers or diffusers when reflections hide the defect.
• Control exposure and white balance and document camera settings. Use the same settings across the pilot where feasible.
• Capture images across expected shifts if lighting varies (daylight vs artificial) so models learn real conditions or so you can scope hardware solutions.

Labeling conventions and formats

Define a single, simple annotation schema before labeling begins and document it.

• Annotation type: classification (image‑level), bounding box, segmentation mask, keypoints—choose the simplest form that supports the acceptance criteria. For small surface defects, segmentation or high‑precision bounding boxes are usually best.
• Label taxonomy: provide exact class names, codes, and examples. Include an explicit "uncertain/ambiguous" or "inspect" label policy to avoid inconsistent labeling.
• File formats and standards: store annotations in a common open format such as COCO, Pascal VOC, or a documented CSV manifest. Save a dataset manifest with sample metadata (camera ID, timestamp, operator, lot number, fixturing flag).
• Naming and versioning: use stable filenames and a versioned dataset prefix (e.g., dataset_v1a_manifest.json). Never overwrite labeled data in place without versioning.

Labeling workflows, QA, and inter‑rater agreement

Label quality matters more than labeling quantity.

• Use a labeling tool that supports review and exports to your chosen format (LabelImg, VGG Image Annotator, Supervisely, CVAT, etc.).
• Create a labeling guide with 15–30 labeled examples per class (good and bad) and edge cases with notes.
• Run inter‑rater agreement checks: have multiple labelers annotate 5–10% of the dataset and compute agreement measures (Cohen’s kappa for classification, IoU overlap for bounding boxes). Target high agreement (kappa > 0.6; IoU > 0.7 for boxes) and iterate the guide if agreement is low.
• Include a QA pass where a subject‑matter expert reviews a random 10% sample for correctness.

Dataset splits and evaluation strategy

• Use a train/validation/test split that prevents leakage. Common splits: 70/15/15 or 80/10/10. Stratify splits by lot, machine, or date to ensure test set reflects true unseen variation.
• Consider time‑based splits for drift detection (train on older data, test on newer batches).
• Reserve a small field validation set for operator A/B testing post‑training—this set should reflect live conditions.

Dataset size and augmentation guidance

There are no universal numbers, but these practical rules help:

• For simple, well‑scoped inspections with strong fixtures and lighting, several hundred labeled defect examples per class can be sufficient. For more variation expect thousands.
• If defects are rare, actively collect targeted defect samples (capture and label every defect for a period) rather than relying solely on random sampling.
• Use augmentation (small rotations, brightness/contrast, slight translations) to increase robustness but avoid augmentations that create unrealistic defects (don’t rotate an asymmetrical part into an impossible view).
• If using synthetic data (rendering or generative augmentation), validate with real data—synthetic can help reduce label cost but rarely replaces all real examples.

Acceptance criteria & evaluation metrics

Make metrics operationally meaningful:

• Primary metrics: recall (sensitivity) for safety/defect capture; precision for minimizing false rejects. Use F1 as a balanced metric when appropriate.
• Set thresholds tied to business outcomes (e.g., reduce manual inspection load by X% while keeping missed defects below Y per 10k parts).
• Also measure inference time, memory footprint, and model stability across lighting and part variation.
• Run a small live validation: run the model in parallel with human inspectors for one shift and log disagreements for root cause analysis.

Dataset health checklist (pilot pre‑training)

• Target classes and acceptance criteria documented and agreed.
• Camera settings, lens, and lighting documented and stable.
• Dataset manifest with metadata exists and is versioned.
• At least one inter‑rater agreement check completed and issues resolved.
• Train/validation/test split created and stratified by relevant keys.
• Label counts per class reviewed and imbalance mitigation planned.
• Negative/near‑miss examples included.
• Annotation format chosen and sample export validated.

Integration, deployment, and maintenance notes

• Plan integration points early: where will inference run (camera edge, gateway, plant server) and how will results be recorded (MES, PLC, historian)?
• Define change control for model updates and dataset versions; include rollback procedures for failing models.
• Plan for ongoing data capture and labeling of model errors (false positives/negatives) to support continuous improvement.
• Schedule periodic revalidation (after process changes, new suppliers, or seasonal shifts) and tag images with the conditions that caused failures.

Practical tools and formats

Annotation tools: LabelImg, CVAT, VGG Image Annotator, Supervisely, Scale AI tooling. Formats: COCO JSON for detection/segmentation, Pascal VOC XML for boxes, simple CSV manifests for classification. Pick tools that export your chosen format and support QA workflows.

Next steps

Use this guide to collect a first labeled dataset, run an initial training round, and evaluate against the acceptance criteria. Expect several short iterations: fix imaging and fixturing issues first, then relabel edge cases and expand data variety. Keep operators and quality staff involved—operator validation is often the final test for real world usefulness.

Tip: Start small, measure concretely, and make decisions based on the pilot acceptance criteria rather than a desire to keep experimenting indefinitely.