Data Incident Postmortem & Root Cause Runbook

{"Title":"Data Incident Postmortem & Root Cause Runbook","IntroductionHtml":"

Purpose

This runbook helps teams capture a clear, actionable postmortem after a data, analytics, or model incident. Use the form to record what happened, how it was diagnosed and mitigated, the root cause using structured techniques, and the corrective actions with owners and timelines. Structured entries make follow-up easier and enable team learning.

Tips before you start

• Prefer facts: record timestamps, who performed actions, and concrete evidence (logs, query results, dashboards).
• If possible, link to the original incident/ticket and snapshots of relevant metrics or dashboards.
• Keep communications factual and focused on impact, containment, and next steps.

","SubmitLabel":"Save Postmortem","SuccessMessage":"Postmortem saved. You can return to the incident or continue editing.","DataType":"postmortem","SchemaVersion":"1.0","Fields":[{"Key":"incident_id","FieldType":"text","Label":"Incident ID / Ticket","Required":true,"HelpText":"Unique identifier (ticket, alert ID, or internal incident number)."},{"Key":"incident_title","FieldType":"text","Label":"Incident Title","Required":true},{"Key":"reported_by","FieldType":"text","Label":"Reported By (name / team)","HelpText":"Who first reported or observed the incident."},{"Key":"reported_contact","FieldType":"text","Label":"Reporter Contact (email / Slack)","HelpText":"Best contact for clarifying details."},{"Key":"earliest_known_time","FieldType":"text","Label":"Earliest Known Time (UTC/local)","HelpText":"Earliest timestamp when the problem is known to have existed (approximate is OK)."},{"Key":"detection_time","FieldType":"text","Label":"Detection Time","HelpText":"When the problem was detected or alerted."},{"Key":"time_to_detect_minutes","FieldType":"number","Label":"Time to Detect (minutes)","HelpText":"Elapsed minutes between earliest known time and detection."},{"Key":"time_to_recover_minutes","FieldType":"number","Label":"Time to Recover (minutes)","HelpText":"Elapsed minutes from detection to restoration of acceptable service or data quality."},{"Key":"systems_affected","FieldType":"textarea","Label":"Systems / Pipelines Affected","HelpText":"List affected services, pipelines, databases, models, dashboards, APIs, etc."},{"Key":"impact_summary","FieldType":"textarea","Label":"Impact Summary","Required":true,"HelpText":"Concise description of business and user impact (who, what, and degree)."},{"Key":"user_impact_count","FieldType":"number","Label":"Approx. Users / Records Impacted","HelpText":"If known, provide counts (users, transactions, rows)."},{"Key":"severity","FieldType":"select","Label":"Severity / Priority","Required":true,"Options":[{"Value":"P0","Label":"P0 - Critical / Platform down"},{"Value":"P1","Label":"P1 - Major impact"},{"Value":"P2","Label":"P2 - Partial impact"},{"Value":"P3","Label":"P3 - Minor / cosmetic"}],"HelpText":"Use your team’s severity scale."},{"Key":"immediate_actions_taken","FieldType":"textarea","Label":"Immediate Actions & Containment","HelpText":"What was done to contain damage or mitigate impact (commands, rollbacks, feature toggles, temporary fixes). Include timestamps and actor names."},{"Key":"recent_deployments_checked","FieldType":"yesno","Label":"Were recent deployments checked?","HelpText":"Quick check of whether a recent change may be related."},{"Key":"upstream_producer_health_checked","FieldType":"yesno","Label":"Upstream producer / source health checked?","HelpText":"Confirm whether producer systems were healthy or failing."},{"Key":"data_quality_tests_run","FieldType":"textarea","Label":"Data Quality / Validation Tests Run","HelpText":"Which tests ran and what they showed (schema checks, null rates, distribution changes)."},{"Key":"rollback_performed","FieldType":"yesno","Label":"Was a rollback or undo performed?","HelpText":"If yes, describe what was rolled back and when."},{"Key":"evidence_links","FieldType":"textarea","Label":"Links to Logs, Dashboards, Snapshots","HelpText":"URLs to dashboards, alert pages, log queries, artifacts, or saved outputs supporting the investigation."},{"Key":"root_cause_summary","FieldType":"textarea","Label":"Root Cause Summary","HelpText":"One- to three-sentence summary of the root cause once determined. If unknown, state hypotheses."},{"Key":"five_whys_1","FieldType":"text","Label":"5 Whys - Why #1 (Immediate cause)","HelpText":"Start with the immediate cause (what happened?)."},{"Key":"five_whys_2","FieldType":"text","Label":"5 Whys - Why #2","HelpText":"Why did that happen?"},{"Key":"five_whys_3","FieldType":"text","Label":"5 Whys - Why #3","HelpText":"Deeper causal factor."},{"Key":"five_whys_4","FieldType":"text","Label":"5 Whys - Why #4","HelpText":"Deeper causal factor."},{"Key":"five_whys_5","FieldType":"text","Label":"5 Whys - Why #5 (Root)","HelpText":"The deeper organizational, process, or technical weakness uncovered."},{"Key":"causal_tree_notes","FieldType":"textarea","Label":"Causal Tree / Additional Analysis Notes","HelpText":"Optional: sketch of a causal tree, contributing factors, timelines, or test results."},{"Key":"corrective_actions","FieldType":"textarea","Label":"Corrective Actions (description)","HelpText":"Describe recommended fixes to address root cause(s). Prefer specific, testable actions."},{"Key":"action_owners_and_slas","FieldType":"textarea","Label":"Action Owners and SLA (owner — action — due date)","HelpText":"List owners and target completion dates or SLA windows. Use rows like: Team/Person — Action — Due (YYYY-MM-DD)."},{"Key":"follow_up_monitoring","FieldType":"textarea","Label":"Follow-up Monitoring Plan","HelpText":"What to monitor, metrics/thresholds, and for how long (e.g., 7 days of elevated TTL checks)."},{"Key":"monitoring_duration_days","FieldType":"number","Label":"Monitoring Duration (days)","HelpText":"How long enhanced monitoring should run."},{"Key":"communications_sent","FieldType":"textarea","Label":"Communications (what was sent and when)","HelpText":"Copy or summary of customer/internal communications: who, channel, message summary, timestamp."},{"Key":"stakeholders_notified","FieldType":"textarea","Label":"Stakeholders Notified","HelpText":"List teams, managers, or external parties notified about the incident and updates."},{"Key":"postmortem_meeting_date","FieldType":"text","Label":"Postmortem Review Meeting Date","HelpText":"Scheduled review to discuss the postmortem and confirm actions."},{"Key":"lessons_learned","FieldType":"textarea","Label":"Lessons Learned","HelpText":"Short, actionable lessons the team should remember for future prevention or detection."},{"Key":"prevent_reoccurrence_checklist","FieldType":"checkbox","Label":"Suggested Preventive Measures (select all that apply)","Options":[{"Value":"add_data_quality_tests","Label":"Add or strengthen automated data quality tests"},{"Value":"add_alerts","Label":"Add or tune alerts/thresholds"},{"Value":"deployment_gating","Label":"Improve deployment gating and review"},{"Value":"access_controls","Label":"Adjust access controls or permissions"},{"Value":"improve_docs","Label":"Improve runbooks and operational docs"},{"Value":"team_training","Label":"Team training or tabletop exercises"},{"Value":"other","Label":"Other (describe below)"}]},{"Key":"other_prevention_notes","FieldType":"textarea","Label":"Other Prevention Notes","HelpText":"Describe other prevention ideas not listed above."},{"Key":"postmortem_published_date","FieldType":"text","Label":"Postmortem Published Date","HelpText":"When this postmortem was finalized or published."},{"Key":"postmortem_owner","FieldType":"text","Label":"Postmortem Owner (person responsible for follow-ups)","HelpText":"Who will ensure corrective actions are completed."},{"Key":"closed","FieldType":"yesno","Label":"Is the postmortem closed?","HelpText":"Set to yes when follow-ups are complete and monitoring window ended."},{"Key":"free_notes","FieldType":"textarea","Label":"Additional Notes","HelpText":"Any other context, references or next steps."}]}