AI Model Risk & Safety Assessment Template

Purpose and scope

Use this template to assess AI model risk and safety before moving an idea from prototype toward production. The goal is to surface likely harms, reduce bias and privacy exposure, clarify operational failure modes, and define clear monitoring and rollback criteria so teams can make informed decisions. This is not a replacement for formal legal or compliance review.

How to use this template

1. Fill out the Use Case and Context fields with concrete examples and user journeys.
2. Complete the Harm Surface Map to identify where and how the model could cause harm.
3. Assess data provenance, fairness checks, and performance thresholds using the prompts below.
4. Define monitoring metrics, alert thresholds, test cases, and rollback criteria before launching experiments that impact real users or decisions.
5. Store this completed assessment alongside your prototype artifacts, test results, and sign-offs.

1. Use case description

Describe the model and the context in plain language. Attach or link to any diagrams or user flows.

• Model name / version: ________________________
• Purpose / decision supported: ________________________
• Primary users / affected parties: ________________________
• Operating environment (prototype, pilot, offline eval): ________________________
• Primary inputs and outputs: ________________________
• Business, legal or regulatory constraints: ________________________

2. Harm surface map

Map where harms could occur. For each item, describe the harm, who is affected, how likely it is, and how severe the impact would be.

1. Incorrect outputs / false positives/negatives — Affected parties: ______ — Likelihood: (Low/Med/High) — Severity: (Low/Med/High)
2. Privacy leakage (re-identification, PII exposure) — Affected parties: ______ — Likelihood: ____ — Severity: ____
3. Disparate impact (systematically worse outcomes for a protected group) — Groups: ____ — Likelihood: ____ — Severity: ____
4. Adversarial or misuse risks (gaming, data poisoning) — Scenarios: ____ — Likelihood: ____ — Severity: ____
5. Operational hazards (latency causing degraded UX, cascading failures) — Scenarios: ____ — Likelihood: ____ — Severity: ____
6. Reputational harm, regulatory exposure, legal risk — Describe: ____

Guidance: Prioritize harms where both likelihood and severity are medium or high. Capture at least one plausible real-world example for each prioritized harm.

3. Data provenance and quality

Questions to answer and evidence to attach.

• Data sources (list datasets, owners, collection method): ______
• Intended vs. actual population coverage: ______
• Sampling methods and known biases: ______
• Labeling process and quality checks (who labeled, inter-rater reliability): ______
• Data licensing, consent status, retention rules: ______
• Are PII or sensitive attributes present? If so, how are they protected or removed?: ______
• Provenance checklist: source documented, versioned snapshot stored, data lineage trace available (yes/no)

4. Fairness & bias checks

Run these checks and record findings. If you can’t run quantitative checks now, describe planned evaluations and timelines.

• Define protected and relevant groups for this use case: ______
• Performance by group (attach confusion matrices, error rates): ______
• Decision threshold sensitivity by group: ______
• Calibration and score distribution checks: ______
• Statistical parity, equalized odds, or other chosen fairness metrics and results: ______
• Qualitative review for culturally-specific errors or inappropriate content: ______

5. Performance thresholds and acceptance criteria

State measurable thresholds that must be met before advancing the model. Include minimum acceptable values, target values, and how metrics are measured.

• Primary metric(s) (e.g., precision@k, recall, AUC): metric = ______, minimum acceptable = ______, target = ______
• Secondary metrics (robustness under distribution shift, latency, throughput): ______
• Fairness thresholds by group (e.g., max allowed disparity): ______
• Privacy constraints (e.g., differential privacy budget, no retention of raw PII): ______

6. Suggested safety and robustness test cases

Design tests that reflect real-world failure modes. For each test include objective, pass/fail metric, and data used.

1. Adversarial robustness test — Objective: model resists small perturbations; Metric: drop in accuracy <= X% on perturbed set.
2. Out-of-distribution input test — Objective: model flags or degrades gracefully; Metric: confidence calibration / fallback triggered.
3. Bias amplification test — Objective: does the model amplify existing disparities? Metric: compare metric ratios pre/post model.
4. Privacy leakage test — Objective: detect presence of memorized PII; Metric: success rate of extraction attack (should be near 0).
5. Human-in-the-loop handoff simulation — Objective: safe escalation for uncertain or high-risk cases; Metric: handoff accuracy and time to resolution.
6. Monitoring/alert simulation — Objective: ensure alerts fire when thresholds crossed; Metric: alert precision and lead time.

7. Monitoring plan

Define what will be continuously measured, who is notified, and what actions are triggered.

• Key operational metrics: model uptime, latency, error rate, input distribution drift, confidence distribution.
• Key safety metrics: rate of high-severity errors, fairness metric drift, number of privacy incidents.
• Alert thresholds and owners: (metric -> threshold -> notify -> action)
• Frequency of checks (real-time / daily / weekly) and responsible team members.
• Data retention for audit and retraining: location, format, retention period.

8. Rollback and mitigation criteria

Define quantitative and qualitative conditions that will trigger rollback or temporary disablement.

• Immediate rollback triggers (example): fairness disparity > X, critical error rate > Y, confirmed privacy leakage.
• Temporary mitigation options: reduce rollout percentage, disable risky features, route to human review.
• Post-incident steps: triage, root-cause analysis, patch, test, redeploy procedure, and communication plan.

9. Documentation & record requirements

Ensure these items are captured and stored with the assessment.

• Model specification and training recipe (hyperparameters, architecture) — stored (yes/no)
• Training data snapshot and provenance records — stored (yes/no)
• Evaluation datasets and test case results — stored (yes/no)
• Fairness and privacy test results — stored (yes/no)
• Monitoring dashboard links and alert configurations — stored (yes/no)
• Sign-offs: technical owner, product owner, legal/compliance (if required), ethics reviewer — names & dates

10. Decision & next steps

Choose one:

• Proceed to constrained pilot (specify constraints): ______
• Proceed to expanded pilot (specify additional controls): ______
• Defer until changes made (list required changes): ______
• Do not proceed — rationale: ______

Quick checklist (one-page)

• Use case and examples documented
• Harm surface map completed and prioritized
• Data provenance and snapshots saved
• Fairness checks executed and reviewed
• Performance thresholds defined and met
• Test cases executed with results recorded
• Monitoring and alerting defined with owners
• Rollback criteria and mitigation plan defined
• Documentation stored and sign-offs recorded

Appendix: example metrics and thresholds (suggested starters)

• Error-rate increase tolerance: <= 5% relative to baseline
• Group disparity tolerance: disparity ratio <= 1.25
• Alert if input feature distribution shift (population JS divergence) > 0.1
• Latency SLO: p95 <= 300ms for synchronous services
• Privacy: no storage of raw unredacted PII; DP epsilon target (if used): <= 2

Notes and references

Link to company policy, legal guidance, model cards, datasheets, and any prior incident reports. Use this template as a living document: revisit after pilots or incidents.