AI Prototyping Safety & Ethics Checklist

Welcome — How to use this checklist

This checklist helps teams prototype AI ideas quickly while keeping legal, ethical, privacy, and safety risks manageable. Use it as an operational runbook during design, data collection, model training, evaluation, and early testing. It assumes prototypes are not production systems and that prototypes touching sensitive data, safety-critical decisions, or regulated domains will receive additional governance and legal review.

Start with the Quick Risk Triage, follow the sections most relevant to your prototype, record outcomes, and complete a short Model Risk Assessment before any decision to scale or productionize.

Quick Risk Triage

• Data sensitivity: Does the prototype use personal data, health data, financial data, or other regulated / high-risk data? (Yes / No)
• Decision impact: Does the model influence safety, finances, legal status, employment, or access to essential services? (High / Medium / Low)
• Visibility & exposure: Who will see model outputs? Internal teams only, selected users, or public / customer-facing?
• Adversarial risk: Could an attacker manipulate inputs to produce harmful outputs? (High / Medium / Low)
• Human oversight: Is a human-in-the-loop (HITL) required? If yes, what decisions remain with humans?

1. Data provenance & consent

• Record data sources, collection dates, and original purpose for each dataset used.
• Confirm consent or lawful basis for using personal data in the prototype. Keep evidence of consent and retention limits.
• Apply minimal data principle: use only the fields required for the prototype task.
• Note any third-party data licenses and restrictions (e.g., no retraining, no redistribution).
• Maintain a data lineage file: raw data → cleaned data → features → training / test splits.

2. Bias & fairness quick checks

• Define relevant protected attributes and groups for your context (e.g., race, gender, age). Even if not explicitly used, evaluate proxy features.
• Run group performance comparisons on key metrics (accuracy, precision/recall, FPR/FNR) and flag disparities exceeding an agreed threshold.
• Perform simple counterfactual checks (does small, irrelevant change to input change outcome unfairly?).
• Document known dataset skews and sampling biases. Note whether the prototype amplifies or reduces these skews.
• Plan mitigations: reweighting, resampling, constraint-aware training, or post-processing adjustments. Record chosen approach and rationale.

3. Privacy impact

• Conduct a lightweight Privacy Impact Assessment: what personal data is used, how long is it stored, with whom is it shared?
• Anonymize or pseudonymize data where possible. If using raw identifiers, justify necessity.
• Verify encryption-at-rest and in-transit for any stored prototype data or model artifacts.
• Limit access: restrict prototype data and models to a named, small team with logged access.
• If the prototype uses synthetic data or differential privacy, document settings and expected trade-offs.

4. Logging, explainability & audit trail

• Log inputs, outputs, timestamps, model version, and user or system identity for each prototype run (sufficient for debugging and incident review).
• Record seed values and environment (software versions, libraries) used to train/serve models.
• Provide a human-readable explanation for outputs where feasible (feature attributions, rules summary, or exemplar cases).
• Keep a changelog for model changes and dataset updates. Tag artefacts with version IDs.

5. Human-in-the-loop (HITL) boundaries

• Explicitly define which actions are automated and which require human approval.
• Design clear escalation paths and decision thresholds for when humans must intervene.
• Provide human reviewers with sufficient context to interpret model suggestions (confidence scores, salient features, alternatives).
• Train reviewers on common failure modes and how to record overrides or feedback.

6. Performance thresholds & acceptance criteria

• Set objective, measurable criteria for prototype success (e.g., accuracy ≥ X, F1 ≥ Y for target groups, latency < Z ms).
• Define acceptable ranges for fairness metrics and minimum confidence thresholds for outputs shown to users.
• Specify minimum sample sizes for test evaluations and holdout strategies (time-based split, cross-validation as appropriate).
• List stop conditions: do not proceed if any critical privacy, safety, or legal item is unresolved.

7. Monitoring signals (during limited release or pilot)

• User-facing error rates, latency, and throughput.
• Distributional drift indicators for key input features and model outputs.
• Feedback/override rates from human reviewers.
• Adverse event reports, complaint volume, or qualitative user feedback highlighting confusion or harm.
• Unusual spikes in edge-case inputs that correlate with misbehavior.

8. Rollback & incident response plan

• Define immediate rollback steps (e.g., stop serving model version X, switch to safe baseline or manual process).
• Assign incident roles: incident lead, communications lead, technical lead, legal/compliance contact.
• Keep a simple incident log template: time, trigger, impact, actions taken, and follow-up tasks.
• Plan customer or user communications for visible incidents; avoid technical blame and focus on corrective steps and support.
• After incident: run a root-cause review and update this checklist and training/data as appropriate.

9. Short examples for common prototype types

Classification prototype (e.g., triage emails)

• Data: sample labeled set plus synthetic negatives to test edge cases.
• Checks: group performance by sender region and mail language; record false positives that could block important messages.
• HITL: human approves any classification with confidence < 0.7.

Recommendation prototype (e.g., content suggestions)

• Data: click and impression logs, with timestamped context.
• Checks: exposure fairness by user segment; ensure no filter bubble amplification for protected attributes.
• Monitoring: CTR, engagement, diversity metrics, and complaint/feedback rate.

Generative prototype (e.g., text or image generation)

• Data: training or prompt sources flagged for copyrights or sensitive content.
• Checks: toxic content filters, hallucination rate estimation, and provenance tagging for outputs.
• Safety: do not display synthetic content as factual without clear provenance and disclaimers.

10. Model Risk Assessment (lightweight template)

Use this short template to capture the key risk decisions for the prototype. Keep the filled assessment with project notes.

• Project name: __________________________
• Model / Prototype description: One-sentence summary.
• Data sources & sensitivity level: (None / Low / Medium / High) — Note required consents.
• Primary harms to consider: (privacy, discrimination, safety, reputational, financial)
• HITL required: Yes / No — If yes, describe human role.
• Acceptance criteria: Key metrics and thresholds.
• Monitoring signals to track: list top 3–5.
• Rollback plan summary: One-line steps.
• Reviewer & date: __________________________

Practical reminders & governance boundaries

• This checklist does not replace formal legal, compliance, security, or clinical reviews when required. Escalate as needed.
• Label prototypes clearly in documentation and UIs when they are experimental.
• Do not treat prototype results as production-grade without additional validation, load testing, and governance sign-off.

Next steps (if prototype meets criteria)

1. Complete the Model Risk Assessment and store it with project artifacts.
2. Conduct a controlled pilot with monitoring enabled and a predefined observation window.
3. Gather quantitative and qualitative feedback, update data and models, and re-evaluate against acceptance criteria.
4. If moving toward production, prepare production-grade controls: hardened access, formal privacy review, security testing, SLA planning, and operational monitoring.