Vision Labeling Template & Labeling SOP

Purpose

Provide a concise, enforceable labeling standard so all annotators produce consistent ground truth for visual-inspection AI pilots. Use this SOP to prepare datasets, train labelers, run pilot labeling, perform QA, and export production-ready annotation manifests.

Scope

Applies to image-based visual inspection pilots (bounding box, segmentation/masks, and keypoint annotations) for defects, components, or part features. Adapt for video frames using the same rules per frame.

Quick Checklist (use at start/end of each labeling session)

• Load labeling schema version: v{major.minor}
• Confirm annotator training completed and quiz passed
• Verify tool settings (zoom, brush size, snapping) match this SOP
• Complete pilot batch of N images for QA review
• Run export checklist and validate file schema and sample labels

Label Naming Conventions

• Use lowercase, snake_case for label names: surface_crack, missing_fastener.
• Prefer noun phrases that describe the defect or object, not action verbs: dent not dented.
• Where hierarchical categories exist, use a dot-separated hierarchy in metadata (not as the display label): category.subcategory (e.g., surface.crack).
• Keep label names stable. If changing, increment schema version and document mapping from old to new labels.

Label Types & When to Use Them

• Bounding box: use when defect is well localized but pixel-precise boundaries are unnecessary (fast annotation, object detection).
• Segmentation mask / polygon: use when exact shape matters for severity or measurement (area-based defects, coatings, spall).
• Keypoints: use for landmarks or alignment points (bolt center, hole location).
• Attributes: attach modifiers to labels (e.g., severity:low/medium/high, confirmed_by:operator/qa).

Annotation Rules (concrete)

1. Draw the smallest shape that fully contains the defect without including adjacent unrelated features.
2. For masks, follow the true visible edge of the defect; exclude reflections or glare that do not correspond to material change.
3. When multiple instances of the same defect appear, label each instance separately.
4. For grouped defects (many small scratches clustered), choose either (a) a single polygon covering the cluster with attribute cluster=true, or (b) label each scratch individually—document which approach is used for the dataset split.
5. If an object is partially outside the frame, annotate only the visible portion and mark attribute partial=true.
6. Do not label features that are not relevant to the pilot objective (e.g., background marks, lens dirt) unless explicitly included in schema.

Positive / Negative Examples (describe—use image references in practical use)

For each label include three positive and three negative example images (link or attach images in the project workspace). Examples should illustrate typical, subtle, and difficult cases. Negative examples should show similar-looking items that should NOT be labeled so annotators learn the boundary decisions.

Edge-case Guidance

• Occlusion: If a defect is partially occluded but still recognizable, label the visible portion and add attribute occluded=true. If recognition is uncertain, flag for reviewer.
• Reflection and glare: If reflection alters appearance but not material condition, do not label as defect. If glare hides a defect, mark as uncertain and escalate to QA.
• Wear vs. defect: Define threshold (e.g., >X mm depth or >Y% area) when wear counts as a defect. Include measurement examples.
• Overlapping defects: Label each defect separately even when overlapping; use z-order attribute if order matters.

Annotator Workflow (SOP Steps)

1. Read the current labeling schema and view the linked example gallery.
2. Complete the training batch (N labeled images) and pass internal QA quiz.
3. Label images using the assigned label tool and save progress frequently.
4. Mark items you are unsure about with a 'flag_for_review' tag.
5. When batch complete, submit for reviewer QA using the project review queue.

Reviewer / QA Process

• Perform batch QA on a random sample (suggest 10–20% of annotator output) and a focused sample of flagged items.
• Measure inter-annotator agreement (IoU for boxes/masks, label agreement for classes). Target: mean IoU >= 0.6 for boxes; class agreement >= 90% depending on task.
• Use confusion matrix to identify ambiguous labels and update schema/examples when systematic confusion arises.
• Provide annotated feedback to labelers and require rework when agreement thresholds are not met.
• Keep a QA log entry per batch describing common errors and schema clarifications.

Acceptance Criteria for Pilot Labels

• Sample pass rate: >= 90% class agreement on QA sample
• Localization: mean IoU for defects >= 0.6 (adjust per use case; document chosen threshold)
• All flagged items resolved or explicitly marked uncertain with documented reason

Label Export Checklist (before model training)

1. Export format: specify (COCO JSON, Pascal VOC XML, YOLO TXT, custom CSV). Include schema example and version.
2. Required fields for each annotation: image_id, label, bbox[x,y,w,h] or segmentation coordinates, confidence (if applicable), attributes (severity, partial, occluded), annotator_id, created_date.
3. Include a separate label_map file mapping label names to numeric IDs.
4. Verify a representative subset of exported annotations against original images.
5. Create train/val/test split at image level (not per-annotation) and document split ratios and random seed.
6. Record dataset manifest with counts per label, per-split distributions, and QA pass rates.

Versioning and Change Control

• Every modification to the schema requires a new schema version and a migration mapping from prior versions.
• Track who approved the change, reason, and date in the dataset manifest.

Role Definitions

• Annotator: applies labels per schema.
• Reviewer / QA: samples and approves annotated batches, records issues.
• Dataset Owner: approves schema changes and signs off on export for modeling.

Operational Notes & Handover

Attach the labeling tool project link, example gallery, current schema JSON, label_map, and a QA log when handing over data for training. Preserve original images and exports in a versioned dataset bucket with clear naming: project_schema_v{major.minor}_export_{YYYYMMDD}.zip

References & Templates to Attach

• Labeling schema JSON template (fields documented)
• Export manifest template (CSV/JSON)
• QA log template
• Example gallery with positive / negative / edge cases