Data Incident Response Playbook

Purpose & Scope

This playbook helps teams respond to data incidents in production: data corruption, accidental deletion, incorrect schema changes, mass mis-labeling, data pipeline failures that change business outputs, or suspicious data access. Use this runbook for operational response and learning. It does not replace legal, compliance, or forensic investigations — escalate to specialists when required.

Quick start (first 10–30 minutes)

1. Identify an Incident Commander (IC) and announce a dedicated incident Slack channel.
2. Triage severity (see severity matrix below) and classify impacted systems and data domains.
3. Contain impact to prevent further damage (freeze pipelines, revoke risky jobs, disable writes where safe).
4. Preserve evidence (snapshots, logs, config, query history) before any destructive action.
5. Begin stakeholder communication: internal triage update and customer-facing ack if needed.

Roles & Responsibilities

• Incident Commander (IC): Overall coordination, scope, severity, and communications owner.
• Data Owner: Domain expert who clarifies business impact and validates recovery decisions.
• SRE / DBA / Pipeline Owner: Technical lead for containment, rollback, and recovery steps.
• Security / Compliance: Assess unauthorized access, data breaches, and legal obligations.
• Communications / Support: Draft internal & external messages, coordinate customer-facing notices.
• QA / Validation: Define tests to confirm recovery correctness and data integrity.

Incident Severity Matrix

Use severity to decide responders, escalation, and communication cadence.

Detection & Classification

1. Record detection source (alert, customer report, monitor, audit, deploy).
2. Capture initial facts: time discovered, last known-good timestamp, data sources/tables/streams involved, affected services and customers.
3. Classify incident type: corruption, deletion, incorrect transformation, schema drift, unauthorized access, latency/backpressure.

Immediate Mitigation & Containment

• Stop onward writes to affected data stores where safe.
• Pause scheduled jobs and suppress automated downstream consumers (dashboards, jobs) to avoid spreading bad data.
• Enable read-only access where possible to allow diagnostics without further changes.
• Take immutable snapshots and collect logs, query history, access logs, job run metadata, and recent backups.

Rollback vs. Patch Decision Tree (summary)

Use this checklist to decide whether to rollback data to a known good state or apply a corrective patch/transformation:

1. Can we identify a recent, complete, and validated backup/snapshot that predates the incident?
2. Would restoring that backup return business logic to correct behavior without unacceptable data loss (e.g., loss of post-backup legitimate updates)?
3. Can a targeted, idempotent repair (patch) fix the affected subset with lower risk than full rollback?
4. Are there downstream consumers that must be coordinated (billing, analytics, ML models)?
5. Is there regulatory/legal risk from restoring or modifying records?

If most answers favor a safer, targeted repair and verification tests exist, prefer patch. If corruption is broad, inconsistent, or unknown, prefer rollback to a validated snapshot and replay valid events where possible.

Recovery & Validation

1. Execute chosen recovery (rollback restore or corrective transformation) in a staging environment first and run the full validation suite.
2. Validation should include record counts, checksums, business metric reconciliation, sample customer verification, and model sanity checks when ML outputs depend on data.
3. Gradually re-enable downstream consumers and monitor for anomalies; use canary rollouts and short feedback loops.

Stakeholder Communications (templates below)

Use the incident channel for technical coordination. Use defined cadence (e.g., every 30–60 minutes for P0) for updates. Prepare a customer-facing acknowledgement for high-impact incidents.

Templates

Slack — Internal (short)

  [INCIDENT] [P1/P0] | Incident Commander: @alice | Affected: billing-db (writes); Start: 2026-08-26T12:15Z
  Action: paused ETL jobs, taking snapshots. Next update in 30m. IC channel: #incident-data-123
  

Email / Customer-Facing Acknowledgement

  Subject: Service Notice — Data Issue Affecting [Feature/Customers]

  Hi [Customer],

  We’ve detected an issue affecting [brief description of impact]. Our team has identified the cause and is working to contain and repair the data. We’ve paused the affected processes to prevent further impact and will provide an update within [time window].

  We apologize for the inconvenience. If this affects billing or critical workflows, please reply to this message and we will prioritize your case.

  — [Company] Ops
  

Evidence Preservation Checklist

• Export and store snapshots/backups with timestamps and checksums.
• Save pipeline run IDs, job logs, config commits (git SHA), deployment IDs, and recent queries.
• Collect access logs and account actions around the incident window.
• Record exact commands or scripts used for any emergency correction.

Post‑Incident RCA & Prevention

1. Stabilize and close operational tasks (recovery verified, customers notified).
2. Schedule a blameless postmortem within 48–72 hours with the IC, Data Owner, SRE, and any affected stakeholders.
3. RCA agenda: timeline of events, root cause analysis (5 Whys or causal factor chart), contributing conditions, detection gaps, response gaps, and action items with owners and due dates.
4. Define measurable prevention actions: monitoring alerts, improved backups, schema change controls, access restrictions, automated validation gates, and runbook updates.
5. Track action items in a visible tracker and verify completion with post-implementation testing and dashboards showing the metrics used to validate the fix.

When to Escalate to Forensics / Legal

• Evidence of unauthorized access or data exfiltration.
• Personal Data loss that may trigger regulatory notification requirements.
• Potential criminal activity or incidents involving law enforcement.

Incident Close Checklist

• All remediation work completed and validated by QA.
• Customers affected have been notified with remediation details.
• RCA meeting held and action items assigned with deadlines.
• Runbooks and monitoring updated to reflect lessons learned.
• Post-implementation verification dashboards created and monitored for the next appropriate window.

Appendix — Useful Queries & Commands (examples)

Include safe, org-specific SQL queries to check counts, checksums, and top offending keys; commands to snapshot replicas; and documented rollback procedures. Keep these in secure internal runbooks and maintain them as part of the domain toolkit.

Note: This playbook is meant to be tailored to your stack (data lake, streaming, OLTP, warehouses) and operational policies. Adapt the rollback procedures and validation suites to match your systems and compliance needs.