Checklist

Data Labeling & Quality Checklist for AI

Interactive checklist to assess dataset readiness, labeling quality, traceability, and bias mitigation. Use to create auditable dataset records that support reproducible, trustworthy AI experiments.

Interactive Tool

Data Labeling & Quality Checklist for AI

Use this checklist to assess dataset readiness, labeling quality, traceability, and bias mitigation. Save responses to create an auditable dataset record that helps teams decide whether to continue, pivot, or stop an AI experiment.

Human-readable name for the dataset (project or repo).
Version string, tag, or commit/hash.
Person, role, or team responsible for curation.
YYYY-MM-DD or similar.
Are instructions, examples, and edge-case rules written down and accessible?
Link or path to the guidelines (DOC, repo, ticket).
Have labelers been trained and assessed for consistency?
Was agreement quantified (e.g., Cohen's kappa, Fleiss' kappa)?
Enter kappa (e.g., 0.75) or percent (e.g., 75). Leave blank if not measured.
Are ambiguous or unusual cases described with examples and decisions?
Describe representative edge cases and the chosen labeling approach.
Has class distribution been measured and recorded?
Provide counts or percentages for major classes (or a short table).
Does the dataset include synthetic or generated examples?
Describe methods, parameters, and rationale for augmentation; note effects on distribution.
Are fields such as source, timestamp, device, annotator, split, and provenance recorded?
List metadata fields captured for each example (e.g., source, annotator_id, timestamp).
Does the dataset contain PII, health data, or other sensitive information?
Were steps taken to remove, redact, or otherwise protect personal data?
Describe removal, hashing, redaction, or other techniques used.
Are usage rights, consent, and licenses documented and compatible with project use?
Is there a manifest, version history, and source tracking for the dataset?
Provide links or notes about data sources, ingestion processes, and version control.
Were checks for duplicates, corrupted files, label leakage, distribution shifts, and annotation drift performed?
Summarize the checks performed and any findings; include scripts or ticket refs where possible.
Any observed demographic, sampling, or label biases?
Actions taken or planned to reduce bias (sampling, relabel, weighting, augmentation).
How to reproduce dataset split, preprocessing, and label logic (commands, scripts, seed values).
Team judgement about dataset readiness for model training.
How confident are you in this checklist today? 1=low, 5=high
1.0 10.0
Additional notes, links to manifests, tickets, or next steps.
You can explore this tool now. Sign in or create an account to save your responses and return to them later.
Sign In to Save Create an Account
Make this tool part of your work

Save a personal copy, bring it to your team, or tailor the questions and workflow to fit what you are hungry to improve.

Member customization and team collaboration are coming soon.
Discuss Ask the Librarian

Discussion

Comments and conversation will live here.