Industrial AI Pilot Evaluation Checklist (MLOps-lite)

{"Title":"Industrial AI Pilot Evaluation Checklist (MLOps-lite)","IntroductionHtml":"

Purpose: A short, operational checklist to evaluate whether an industrial AI pilot is ready to scale or needs further controls. Use this during pilot reviews to ensure teams consider safety, reliability, operator workflows, monitoring, rollback, explainability, and ownership—not just accuracy numbers.

How to use: Complete required fields, attach links to validation artifacts, and record the reviewer, outcome, and recommended actions. Saved responses create an audit trail for governance, audits, and post-mortems.

","SubmitLabel":"Save evaluation","SuccessMessage":"Evaluation saved. Use the saved responses for audits, follow-ups, or to seed a deployment plan.","DataType":"IndustrialAIPilotEvaluation","SchemaVersion":"1.0","Fields":[{"Key":"baseline_metric","Type":"textarea","Label":"Baseline metric(s) and current value(s)","HelpText":"E.g., OEE=72%, false alarm rate=3%, mean time between failures (MTBF). List metric name and baseline numeric value.","Required":true},{"Key":"validation_set_independence","Type":"yesno","Label":"Is the validation/test set independent of training data?","HelpText":"Confirm there is no data leakage or label reuse between training and validation/testing.","Required":true},{"Key":"expected_uplift_vs_cost","Type":"textarea","Label":"Expected uplift vs. cost and acceptance criteria","HelpText":"Describe expected measurable improvement, minimum acceptable uplift, anticipated savings or cost impact, and the acceptance criteria for scaling.","Required":true},{"Key":"monitoring_plan","Type":"textarea","Label":"Monitoring plan for performance and data drift","HelpText":"List metrics to monitor, sampling frequency, drift detection methods, alert thresholds, and who receives alerts.","Required":true},{"Key":"rollback_plan","Type":"textarea","Label":"Rollback and mitigation plan","HelpText":"Describe the steps to safely revert to the previous behavior, how to validate rollback success, and fallback operator actions.","Required":true},{"Key":"human_in_the_loop_rules","Type":"textarea","Label":"Human-in-the-loop rules and operator responsibilities","HelpText":"Define when operators must intervene, decision authority, override procedures, and training required for responders.","Required":true},{"Key":"latency_throughput_constraints","Type":"textarea","Label":"Latency & throughput constraints and test evidence","HelpText":"Specify acceptable response times, throughput limits, test results, and evidence that real workloads meet these constraints.","Required":true},{"Key":"ownership_model","Type":"textarea","Label":"Ownership and maintenance responsibilities","HelpText":"Identify the owner for model updates, data stewardship, monitoring, incident response, and governance artifact custody.","Required":true},{"Key":"explainability_notes","Type":"textarea","Label":"Explainability / interpretability evidence","HelpText":"Summarize what explanations are available to operators (feature importances, rule lists, example cases) and any limitations.","Required":false},{"Key":"safety_risk_level","Type":"select","Label":"Estimated safety/compliance risk level if model fails","Options":[{"Value":"low","Label":"Low"},{"Value":"medium","Label":"Medium"},{"Value":"high","Label":"High"}],"Required":true},{"Key":"evidence_links","Type":"textarea","Label":"Links to evidence and artifacts (validation reports, dashboards, test cases)","HelpText":"Provide URLs, document IDs, or storage locations for replay tests, validation results, and dashboards.","Required":false},{"Key":"confidence_scale","Type":"scale","Label":"Reviewer confidence in readiness (1 = low, 5 = high)","HelpText":"Rate how confident you are that this pilot can be safely scaled with the specified controls.","Required":true},{"Key":"evaluation_outcome","Type":"select","Label":"Recommendation","Options":[{"Value":"proceed_scale","Label":"Proceed to scale with controls"},{"Value":"proceed_limited","Label":"Proceed to limited deployment / extend pilot"},{"Value":"hold","Label":"Hold — revise and retest"},{"Value":"rollback","Label":"Rollback / do not deploy"}],"Required":true},{"Key":"required_followups","Type":"textarea","Label":"Required follow-up actions and owners","HelpText":"If not green, list required remediation items, owners, and target dates.","Required":false},{"Key":"reviewer_name","Type":"text","Label":"Reviewer name","Required":true},{"Key":"review_date","Type":"text","Label":"Review date","HelpText":"YYYY-MM-DD","Required":true},{"Key":"additional_notes","Type":"textarea","Label":"Additional notes","Required":false}]}