Data Incident Response Runbook

Purpose

This runbook helps teams detect, classify, coordinate, and resolve incidents that degrade data quality, break pipelines, corrupt models, or otherwise impair analytics and decisioning. It focuses on rapid containment, restoring reliable data and services, protecting downstream consumers, and capturing learning to prevent recurrence.

Scope & Roles

Applies to data pipelines, feature stores, model training and serving, ETL/ELT jobs, data marts, reporting layers, and scheduled data deliveries.

• Incident Lead: Owns coordination, status updates, and runbook execution.
• On-call Engineer / Data Owner: Performs technical triage, mitigation, and rollback.
• Business Liaison / Product Owner: Communicates impact to stakeholders and prioritizes consumer needs.
• SRE / Platform: Supports infrastructure-level fixes and access to logs/metrics.
• QA / Data Steward: Validates data correctness and signs off recovery.

Severity Levels & Classification

Classify the incident quickly to guide response scope.

• P0 / Critical: Core pipeline or model down; major business processes blocked; material financial, legal, or safety impact.
• P1 / High: Significant data quality issues affecting many users or important reports; degraded model performance with business impact.
• P2 / Medium: Localized or intermittent data errors, noncritical reports impacted, or degraded but tolerable model behavior.
• P3 / Low: Minor anomalies, exploratory datasets affected, or cosmetic issues with small user impact.

Detection & Initial Classification

1. Record incident start time and assign Incident ID (e.g. DATA-2026-045).
2. Identify detection source: alert, customer report, monitoring dashboard, or scheduled checks.
3. Quickly assess likely scope: affected datasets, jobs, services, and users.
4. Assign Severity (P0–P3) based on consumer impact and business risk.
5. Open a single canonical incident channel (Slack/Teams + incident doc) and declare Incident Lead.

Initial Triage Checklist (first 15–30 minutes)

• Confirm and reproduce the symptom on a representative dataset or job run.
• Capture key evidence: job logs, sample records, schema diffs, recent deployments, config changes, and metrics around the time of onset.
• Identify earliest bad data timestamp and the most recent good timestamp.
• Determine direct downstream consumers (dashboards, reports, ML features, downstream pipelines).
• Decide containment strategy: pause downstream jobs, disable model scoring, or mark datasets as suspect.
• If P0/P1, start regular status updates (every 15–30 minutes) and escalate to stakeholders per the escalation matrix.

Impacted Consumers Registry (template)

Maintain a live list in the incident doc with these columns:

• Consumer Name (product/report/team)
• Contact / Business Liaison
• Impact (blocked, delayed, incorrect)
• Business Priority (High/Med/Low)
• Mitigation applied (paused, fallback, manual fix)
• Owner for validation / sign-off

Temporary Mitigation Patterns (select and adapt)

• Pause downstream jobs: Stop scheduled jobs consuming suspect data to prevent propagation.
• Feature freeze: Stop model scoring or ML feature refreshes until data is validated.
• Switch to last known-good snapshot: Restore a previous dataset snapshot for critical consumers.
• Provide a manual data extract: Deliver corrected CSV/Parquet to key consumers while pipeline is fixed.
• Apply schema mapping or type-cast patch: If a schema drift caused failure, add guarded transforms as temporary patches.

Rollback & Re-run Procedures

1. Verify there is a tested rollback plan for the affected job/model.
2. If rolling back code or config, do it in a controlled environment (staging first if possible) and document changes.
3. For data rollbacks, identify most recent clean snapshot or checkpoint and validate integrity before promoting to production.
4. Re-run affected ETL/ELT jobs on the restored inputs. Use deterministic runs and preserve run IDs for auditability.
5. Run a validation checklist after re-run: row counts, null rates, sample value checks, basic aggregates, and key business KPIs comparisons.
6. Only after data steward and business liaison sign-off, unpause downstream consumers and resume normal cadence.

Communication Templates

Keep messages concise, factual, and include the Incident ID, scope, actions taken, and expected next update time.

Engineer -> Internal Status (example)

Subject: [INCIDENT DATA-2026-045] P1 – Missing rows in orders dataset

Body: We detected missing rows in schema.orders from 2026-08-25 02:00 to 04:00. Impact: BI dashboards and revenue reports show undercount. Actions: paused downstream jobs at 08:45, started reprocessing window 01:50–04:10. Next update: 09:15. Owner: @data_engineer

Incident Lead -> Business Stakeholders (example)

Subject: [INCIDENT DATA-2026-045] Operational impact to revenue reporting

Body: We are investigating a data issue affecting revenue reports for 02:00–04:00 UTC. We have paused downstream refreshes and are reprocessing the source. Estimated time to resolution: 60–90 minutes. We will send an update at 09:15. If you need a manual extract, contact @biz_liaison.

RCA & Learning Capture Template

Capture these fields in the incident document during and after the event:

• Incident ID:
• Summary: One-sentence description of what happened and immediate impact.
• Timeline: Key timestamps (detected, mitigated, recovered, closed) with actions.
• Root Cause / Contributing Factors: Technical cause and human/process/environmental contributors.
• Immediate Fix / Mitigation: What restored service or limited damage.
• Long-term Fixes: Code changes, monitoring, tests, process updates, or training needed.
• Owner & Due Date for Long-term Fixes:
• Prevention & Detection Improvements: New alerts, checks, or runbook updates.
• Lessons Learned: Short bullet list for the team and business.

Postmortem & Follow-up

1. Draft postmortem within 72 hours; share with stakeholders and invite constructive review.
2. Assign owners and target dates for long-term fixes; track in a public backlog.
3. Update monitoring and add automated validation checks (schema checks, row-count alerts, cardinality tests, distribution checks, feature-drift monitors) as needed.
4. Schedule a short blameless review meeting focused on prevention and measurable actions.

When to Close an Incident

• Data restored and validated by Data Steward.
• All critical downstream consumers informed and functioning normally.
• Temporary mitigations removed or converted into monitored, documented changes.
• Postmortem draft created and owners assigned for fixes.

KPIs & Signals to Monitor

• Job success rates and latency
• Schema change events and schema drift metrics
• Row-count and cardinality comparisons to historical baselines
• Model performance vs expected KPIs (AUC, error rates, prediction distributions)
• Consumer-side anomaly reports and dashboard error rates

Quick Reference Links

• Canonical incident doc template: /incidents/template
• Rollback checklist for ETL jobs: /runbooks/etl-rollback
• Monitoring & alerting playbook: /monitoring/data-alerts
• On-call schedule: /oncall/schedule

Notes on Use

This runbook is a practical starting point. Teams should copy it into a live incident template and adapt severity thresholds, communication cadence, and rollback mechanics to their environment. Where possible, convert the "Impacted Consumers Registry", the initial triage checklist, and the RCA capture into interactive forms to preserve incident data, make post-incident analysis easier, and reduce manual transcription errors.