Responsible LLMOps Checklist for Analytics

{ "Title": "Responsible LLMOps Checklist for Analytics", "IntroductionHtml": "

Responsible LLMOps Checklist

This interactive checklist helps analytics teams, data engineers, product owners, and compliance partners evaluate LLM integrations before deployment and during operation. Use it to capture evidence, surface gaps, and produce an auditable record that supports go/no-go decisions and mitigation tracking.

How to use: Complete fields with the appropriate cross-functional reviewers. Provide concise evidence in the text fields. Use the risk score and final recommendation to prioritize mitigations. Save the checklist so the record is stored with the project.

", "DataType": "ResponsibleLLMOpsChecklistResponse", "SchemaVersion": "1.0", "SubmitLabel": "Save Checklist", "SuccessMessage": "Checklist saved. Use this record for audits, follow-ups, and continuous improvement.", "Fields": [ {"Key": "project_name", "FieldType": "text", "Label": "Project / System Name", "HelpText": "Name or identifier for the analytics workflow or product using the LLM.", "Required": true}, {"Key": "review_date", "FieldType": "text", "Label": "Review Date", "HelpText": "Date of this review (YYYY-MM-DD)", "Required": true}, {"Key": "owner", "FieldType": "text", "Label": "Owner / Responsible Team", "HelpText": "Team or person accountable for the LLM integration."}, {"Key": "deployment_stage", "FieldType": "select", "Label": "Deployment Stage", "HelpText": "Choose the current stage.", "Options": [{"Value": "prototype", "Label": "Prototype / Experiment"}, {"Value": "staging", "Label": "Staging / Pre-production"}, {"Value": "production", "Label": "Production"}]}, {"Key": "suitability", "FieldType": "radio", "Label": "Is an LLM the right tool for this task?", "HelpText": "Consider accuracy needs, interpretability, and whether deterministic rules or simpler models would suffice.", "Options": [{"Value": "yes", "Label": "Yes"}, {"Value": "partial", "Label": "Partially / With constraints"}, {"Value": "no", "Label": "No - consider alternatives"}]}, {"Key": "suitability_evidence", "FieldType": "textarea", "Label": "Evidence / Notes on Suitability", "HelpText": "Explain why an LLM is appropriate or what constraints apply."}, {"Key": "data_minimization", "FieldType": "yesno", "Label": "Is data minimization in place?", "HelpText": "Inputs are limited to necessary data and sensitive fields are redacted before sending."}, {"Key": "data_minimization_evidence", "FieldType": "textarea", "Label": "Evidence / How data is minimized", "HelpText": "Describe redaction, hashing, pseudonymization, or schema filters applied."}, {"Key": "data_classification", "FieldType": "select", "Label": "Does the data include sensitive information?", "HelpText": "If yes, note classification and protections.", "Options": [{"Value": "none", "Label": "No sensitive data"}, {"Value": "pseudonymized", "Label": "Pseudonymized / tokenized"}, {"Value": "sensitive_encrypted", "Label": "Sensitive - encrypted in transit & at rest"}, {"Value": "phi_pii", "Label": "Contains PHI/PII or other regulated data"}]}, {"Key": "prompt_safety", "FieldType": "yesno", "Label": "Are prompts sanitized to avoid leaking secrets or unnecessary context?", "HelpText": "Includes removing internal IDs, secrets, or customer data not needed for the task."}, {"Key": "prompt_safety_evidence", "FieldType": "textarea", "Label": "Prompt sanitization approach", "HelpText": "Describe filters, regex rules, or middleware used."}, {"Key": "access_controls", "FieldType": "yesno", "Label": "Are access controls and permissions configured for model use and logs?", "HelpText": "Least privilege applied to APIs, keys, and model management."}, {"Key": "access_controls_evidence", "FieldType": "textarea", "Label": "Access controls details", "HelpText": "List roles, groups, vault usage, or key rotation policies."}, {"Key": "logging_observability", "FieldType": "yesno", "Label": "Is structured logging in place for prompts, responses, costs and errors?", "HelpText": "Ensure logs do not contain raw sensitive inputs unless necessary and protected."}, {"Key": "logging_storage", "FieldType": "text", "Label": "Where logs are stored and retention policy", "HelpText": "E.g., secure logging service, retention days."}, {"Key": "output_validation", "FieldType": "yesno", "Label": "Is output validated before use in analytics or decision-making?", "HelpText": "Includes schema checks, business-rule validation, and anomaly detection."}, {"Key": "fallback_strategy", "FieldType": "textarea", "Label": "Fallback and escalation plan for uncertain or hallucinated outputs", "HelpText": "Describe human review, automated rejection, or conservative defaults."}, {"Key": "human_in_loop", "FieldType": "select", "Label": "Human-in-the-loop controls", "HelpText": "Select the level of human oversight.", "Options": [{"Value": "none", "Label": "None (automated)"}, {"Value": "sampling", "Label": "Sampling / periodic review"}, {"Value": "approval_required", "Label": "Approval required for outputs affecting decisions"}]}, {"Key": "monitoring_metrics", "FieldType": "textarea", "Label": "Monitoring metrics and thresholds", "HelpText": "List metrics (e.g., hallucination rate, confidence score, latency, cost) and alert thresholds."}, {"Key": "cost_controls", "FieldType": "yesno", "Label": "Are cost monitoring and budget controls implemented?", "HelpText": "Token limits, rate limits, budget alerts, and optimization strategies."}, {"Key": "estimated_costs", "FieldType": "number", "Label": "Estimated monthly model cost (USD)", "HelpText": "Approximate run cost to help owners assess economic risk."}, {"Key": "privacy_legal_review", "FieldType": "yesno", "Label": "Has privacy and legal review been completed?", "HelpText": "Includes contracts with model providers, DPAs, or regulatory approvals."}, {"Key": "privacy_legal_evidence", "FieldType": "textarea", "Label": "Privacy / legal approvals and notes", "HelpText": "Describe approvals, open issues, or required mitigations."}, {"Key": "retraining_drift_plan", "FieldType": "yesno", "Label": "Is there a plan for model retraining, drift detection, and retirement?", "HelpText": "Includes signals that trigger retraining or decommissioning."}, {"Key": "retraining_evidence", "FieldType": "textarea", "Label": "Retraining / drift monitoring approach", "HelpText": "Data pipelines, labels, schedule, and ownership."}, {"Key": "incident_response", "FieldType": "yesno", "Label": "Is an incident response plan defined for model failures or data leaks?", "HelpText": "Contact points, steps, and communications templates."}, {"Key": "incident_contact", "FieldType": "text", "Label": "Incident response contact / escalation path", "HelpText": "Name, role, and contact info."}, {"Key": "governance_approval", "FieldType": "radio", "Label": "Has governance (security/compliance/product) approved this deployment?", "HelpText": "Record current approval state.", "Options": [{"Value": "approved", "Label": "Approved"}, {"Value": "conditional", "Label": "Approved with conditions"}, {"Value": "not_approved", "Label": "Not approved / pending"}]}, {"Key": "risk_score", "FieldType": "scale", "Label": "Overall risk assessment (1 low - 5 high)", "HelpText": "Reviewer subjective risk score based on the answers.", "Min": 1, "Max": 5}, {"Key": "final_recommendation", "FieldType": "radio", "Label": "Final recommendation", "HelpText": "Decision based on checklist evidence.", "Options": [{"Value": "go", "Label": "Go"}, {"Value": "go_with_mitigations", "Label": "Go with mitigations"}, {"Value": "no_go", "Label": "No-go / Rework required"}]}, {"Key": "reviewer_notes", "FieldType": "textarea", "Label": "Reviewer notes and next steps", "HelpText": "Actionable remediation steps, owners, and deadlines."} ] }