Vision Inspection Dataset & Labeling Guide

Welcome — what this guide helps you do

This guide helps manufacturing teams run a focused, low‑risk machine‑vision pilot by creating high‑quality image datasets and label sets that give models the best chance of success. It has practical, actionable guidance for capture setups, labeling choices (bounding box vs segmentation), annotation formats, dataset sizing heuristics, validation rules, QA of labels, and common pitfalls with mitigation tactics. Use this as a working checklist during pilot planning and early data collection.

How to use this guide

Read the capture and labeling sections first. Use the dataset checklist during collection. If you plan to scale this pilot across parts or lines, treat the guidance as a repeatable procedure and capture a representative pilot dataset for each product family or fixturing arrangement.

1) Capture setups (practical rules)

• Control the presentation: Keep part positioning, orientation, and fixturing consistent where possible. If parts are presented in different orientations, capture each as a distinct sub‑scenario and label accordingly.
• Lighting: Use controlled, diffuse lighting. Avoid mixed ambient and spot lighting. For reflective surfaces, consider cross‑polarization or shrouding to reduce glare. Capture examples of the worst realistic lighting conditions you'll accept in production.
• Lenses and resolution: Choose optics that give the smallest defect of interest at least 10–20 pixels across (rule of thumb). Telecentric lenses reduce perspective error for precision measurement tasks.
• Triggering and motion: Synchronize exposure to reduce motion blur. For line scanning or moving conveyors, use strobe lighting or short exposure times. Prefer consistent triggers over hand‑held shots.
• Background and contrast: Use neutral backgrounds or fixtures that match production. Avoid backgrounds that create spurious features the model can overfit to.
• Metadata: Record camera, lens, lighting, exposure, date/time, production lot, operator, and part serial/lot. Store metadata with each image to later analyze domain shifts.

2) Which annotation type to choose

Pick the simplest annotation that supports your pilot goals:

• Bounding boxes: Good for coarse detection or when defect location is approximate. Fast to label and often enough to automate triage.
• Segmentation (polygons/masks): Use when defect shape matters (e.g., crack area, contamination) or when you need pixel‑accurate measurements.
• Keypoints/landmarks: Use for pose, alignment checks, or when you need to reference specific feature positions.
• Classification labels only: If you only need pass/fail per image (e.g., whole‑part reject), image‑level labels may suffice, but spatial localization is lost.

When in doubt, collect a small pilot with both bounding boxes and a few segmented masks to compare model performance and annotation cost.

3) Annotation formats and tools

Choose a standard output format so annotations are portable. Common formats:

• COCO (JSON): popular for detection and segmentation, supports rich metadata.
• PASCAL VOC (XML): common for detection tasks.
• YOLO (TXT): lightweight, used by many real‑time models.

Labeling tools to consider: CVAT, LabelImg, Roboflow, Supervisely, VIA. Use tools that export to your chosen format and support versioning/export history.

4) Sample‑size heuristics (practical guidance)

Rules of thumb (depend heavily on defect variability, background complexity, and model family):

• Common defects / simple background: Start with ~500–1,500 annotated images per class for a reliable baseline with transfer learning.
• Moderately complex tasks: 1,500–5,000 images per class improves robustness.
• Segmentation masks: 200–1,000 masks per defect type often required because masks carry more information but are costlier to create.
• Rare defects: Aim for 100–500 positive instances, supplemented with synthetic augmentation, patched inspection, and active learning strategies.
• Patches vs full images: For very large images, crop defect‑centered patches into a balanced dataset to reduce labeling cost and increase effective sample size.

These are starting points. If possible, run a small experiment and use learning curves to estimate when additional data yields diminishing returns.

5) Train / Validation / Test split rules

• Avoid leakage: Split by production lot, time window, or fixture rather than randomly if images from the same part would appear in multiple sets.
• Common splits: 70/20/10 or 80/10/10 (train/val/test). For very small datasets, consider cross‑validation.
• Stratify: Ensure each split preserves class balance for rare defects or use oversampling strategies in training only.
• Operational test set: Reserve a test set that reflects production variation you expect post‑deployment (different lighting, operators, line speeds).

6) Data augmentation & synthetic data

Use augmentation to expand the effective dataset but keep it realistic:

• Geometric: small rotations, flips (only when physically plausible), crops, scaling.
• Photometric: brightness/contrast jitter, blur, noise, and color shifts reflecting real lighting variation.
• Advanced: overlay small synthetic defects onto negatives, GAN‑based augmentation, or physics‑based rendering for underrepresented variations.
• Caveat: synthetic data can help but may introduce a domain gap. Always include real examples in validation and test sets.

7) Label quality assurance

• Label policy document: Build a short, example‑driven labeling guide explaining edge cases and ambiguous conditions.
• Double labeling: Have a random 5–15% subset labeled by two independent annotators to measure agreement (IoU for boxes/masks, kappa for classes).
• Consensus process: Resolve disagreements through a small expert review panel and update the labeling policy to reduce drift.
• Automated QA: Flag labels that are tiny boxes, overlapping contradictory labels, or labels with extreme aspect ratios for review.

8) Acceptance criteria (pilot decision metrics)

Define measurable acceptance criteria before training. Example metrics to track:

• Detection: precision, recall, and F1 at a chosen IoU threshold.
• Segmentation: mean IoU (mIoU) and per‑class IoU.
• Operational: inference latency on target hardware, throughput (parts/hour), false reject rate vs manual inspection, and operator override rate.

Example pilot targets (illustrative — set these to match your safety and customer needs):

• Recall ≥ 0.90 for critical safety defects; tolerate lower precision if an operator reviews flagged parts.
• Inference latency under 200 ms per part for inline inspection; or sustained throughput matching line speed.

Balance precision and recall according to the cost of false accepts vs false rejects. Capture the expected human review capacity in these tradeoffs.

9) Common dataset pitfalls and mitigations

• Poorly controlled lighting: Mitigate with fixtures, strobes, cross‑polarization, or collect representative samples across lighting conditions and include them in the test set.
• Biased sampling: Avoid only collecting easy examples. Intentionally capture hard cases and near misses.
• Labeler drift and ambiguity: Use a short labeling policy and periodic re‑training sessions with annotators.
• Too few positives for rare defects: Use targeted collection (longer capture until you reach target positive counts), patch sampling, and synthetic augmentation combined with active learning.
• Overfitting to background/fixturing: Vary background slightly if production allows or collect multiple fixtures and incorporate them into validation sets.

10) Dataset structure and versioning

Keep datasets organized and versioned:

  dataset-v1/
    images/
      train/
      val/
      test/
    annotations/
      instances_train.json (COCO)
      instances_val.json
      instances_test.json
    metadata.csv  (image filenames, camera, lot, date, labeler)
  

Tag versions when labels change. Use simple semantic versioning (v1, v1.1, v2) and keep changelogs describing added images, relabels, and policy updates.

11) Integration and operational notes

• Validate model predictions in a staging environment connected to MES/PLC before production deployment.
• Design operator workflows for model overrides and feedback capture so flagged mistakes become labeled data for retraining.
• Plan for model maintenance: scheduled re‑labeling, periodic retraining, and monitoring for domain drift (new materials, fixtures, or lighting).
• Include simple health checks: camera focus, lighting intensity, and sample inference checks to detect capture or model degradation early.

12) Quick dataset checklist (printable)

1. Define detection goal and acceptance criteria (precision/recall, latency, throughput).
2. Choose annotation type and format (COCO / VOC / YOLO).
3. Create short labeling policy with examples & edge cases.
4. Plan capture: camera settings, lighting, triggers, fixturing, metadata fields.
5. Collect both typical and edge‑case images; aim for heuristic sample sizes per defect class.
6. Annotate and run label QA (5–15% double labels, review disagreements).
7. Split dataset by lot/time/fixture to avoid leakage; reserve operational test set.
8. Train a baseline model; evaluate against acceptance metrics.
9. Run staged integration tests with MES/PLC and operator review workflow.
10. Version dataset and keep changelog; plan scheduled retraining and monitoring.

Closing guidance

Start small and iterate. A focused pilot with a thoughtfully collected, well‑labeled dataset will reveal whether vision can deliver the value you expect. Use this guide as an operational checklist, adapt the heuristics to your parts and defects, and capture metadata so you can diagnose failure modes quickly.