Data Product Runbook: Health Checks, Incident Playbook & Escalations

Purpose & Scope

This runbook helps data product owners, SREs, analysts, and operators detect problems earlier, respond faster during incidents, restore service reliably, and measure delivery against clear expectations. Use it as a living template: copy it, tailor the indicators and thresholds to your product, integrate with your monitoring, and exercise the procedures regularly.

Product Summary & Consumers (Template)

Keep this section short and current. It helps responders and stakeholders quickly understand the product and who relies on it.

• Product name: [e.g., Customer 360 View]
• Owner / Product Lead: [Name, email, team]
• Primary contacts (ops): [On-call, SRE, data engineer]
• Description: One-line summary of what the product provides.
• Primary consumers: [Teams, dashboards, downstream services, external partners]
• SLAs / SLOs: [Availability, freshness, latency expectations — link to SLA doc]
• Important dependencies: [Sources, pipelines, external APIs, critical tables]
• Runbook version: [vX.Y, last updated date]

Daily & Weekly Health Indicators (Checklist)

Automate these checks where possible. For each indicator, define what triggers an alert and who is notified.

Daily (automated preferred)

• Data freshness: Example — last successful ingest within expected window (e.g., daily pipeline completed < 2 hours late). Threshold: warn at 30m late, critical at 2h late.
• Pipeline success rate: tasks succeeded vs. failed (expectation: 100%). Threshold: warn at 98%, critical at 95%.
• Record counts & deltas: compare expected vs actual volume. Threshold: >10% unexpected variance -> investigate.
• Schema changes detected: new/removed fields in source messages or tables -> alert for inspection.
• Consumer error rate: dashboards or downstream jobs failing due to product data (e.g., >1% error rate -> investigate).

Weekly

• Completeness checks across key dimensions (top 10 customers, regions, product lines).
• Data quality sampling: null rates, value distributions, referential integrity checks.
• Latency and freshness trends (7-day window) to detect regressions.
• Costs and resource anomalies (e.g., runaway queries or storage growth).

Tip: capture these checks as automated jobs that post status to a central dashboard and trigger alerting when thresholds breach.

Incident Classification & Severity Matrix

Agree on a small set of severity levels so responders know how fast to act and what communication is required.

Incident Response Steps (Playbook)

Follow these steps during an incident. Keep action items short and assign a single incident commander (IC).

1. Detect & Acknowledge
   • Alert received from monitoring, consumer, or manual report.
   • IC acknowledges the incident in the incident tracker and sets severity.
2. Assemble
   • Open incident bridge or Slack channel and invite required roles: IC, data engineer, SRE, analyst, product owner, communications.
   • Record time, initial symptoms, and affected consumers/data flows.
3. Quick Triage
   • Is the problem isolated to one pipeline, source, or downstream consumer?
   • Are data values corrupted or missing? Use a lightweight query to confirm.
   • Check recent deployments, schema changes, and external API errors.
4. Containment & Mitigation
   • Apply safe mitigations: pause problematic ingest, re-run last known-good job, switch consumers to a stable cache, or rollback a deployment.
   • Document every change on the incident channel (who, what, why, when).
5. Root-Cause Analysis (RCA)
   • Once service is stable, perform a focused RCA using logs, lineage, and telemetry.
   • Capture causal chain (source -> transform -> storage -> consumer) and identify preventive steps.
6. Recovery & Verification
   • Restore full service, run data quality checks, and confirm consumers can function normally.
   • Validate data integrity (counts, checksums, business-level KPIs) before closing incident.
7. Communications
   • Use consumer communication templates below. Provide status updates at agreed cadences until resolved.
8. Post-Incident Review
   • Schedule a blameless postmortem within 3–7 business days. Produce action items with owners and due dates.

Roles & Responsibilities

• Product Owner: Prioritizes fixes, approves consumer communications, owns SLA changes.
• Incident Commander (IC): Coordinates response, assigns actions, declares resolution.
• Data Engineer / Operator: Executes mitigations, runs replays, applies fixes and rollbacks.
• SRE / Platform: Assists with infra-level problems, scaling, and recovery operations.
• Analyst: Validates data correctness and consumer impact.
• Communications Lead: Crafts messages to consumers and internal stakeholders and ensures cadence.

Consumer Communication Templates

Use these templates and tailor tone and detail level to your audience.

Initial Notification (short)

Subject: [Product] — Incident detected (SEV-[X])
Body: We detected an issue affecting [product name]. Impact: [what consumers see]. We are investigating and will post updates every [interval]. Contact: [support channel/link].

Status Update

Time: [timestamp]
Summary: [current state, actions taken, next steps]
ETA: [estimated time to next update or resolution]

Resolution Notice

Issue resolved at [timestamp]. Root cause: [brief summary]. Actions taken: [replay, rollback, fix]. Consumer impact: [what changed, whether backfill completed]. Postmortem planned: [link or date].

SLA Breach Handling & Measurement

Define clear measurement rules and escalation steps.

• Measurement: use deterministic checks (e.g., pipeline completion timestamp, record-latency percentiles). Store measurements centrally for audit.
• Violation types: Availability, Freshness, Accuracy, Throughput.
• Immediate actions: For SEV-1 SLA breaches, escalate to leadership and trigger root-cause task force. For lower severities, log incident and plan remedial work.
• Reporting: Monthly SLA dashboard that records breaches, MTTR, and recurring failures.

Note: Avoid crafting SLAs that incentivize unsafe behavior (e.g., returning stale data silently to meet freshness SLOs). Prefer observability and consumer-aware fallbacks (e.g., explicit 'stale' flag).

Versioning & Backward Compatibility Guidelines

Protect consumers by managing changes carefully.

• Maintain semantic versioning for published artifacts (APIs, datasets, schemas).
• Apply additive schema changes without breaking consumers; deprecate fields before removal and provide a migration window (e.g., 90 days).
• Provide compatibility tests that run against representative consumer queries.
• Document breaking changes and coordinate releases with consumers using the communication templates and change advisory process.

Deprecation Checklist

1. Announce intent to deprecate with reason and timeline (minimum notice period).
2. List known consumers and impact assessment.
3. Provide migration guidance and sample queries.
4. Monitor consumer adoption; follow-up with owners who haven't migrated as deadline approaches.
5. Only after deadline: disable access, archive data, and publish final notice and postmortem.

Post-Incident Review Template

Keep reviews blameless and action-oriented. Capture:

• Incident summary with timeline
• Root cause analysis
• Immediate mitigations and permanent fixes
• Action items (owner, due date)
• Learnings and test/exercise recommendations

Testing, Exercises & Runbook Governance

• Schedule tabletop exercises twice a year using realistic failure scenarios.
• Run quarterly drills that exercise replays, rollbacks, and consumer communications.
• Review and update the runbook after major incidents, every quarter, or when significant changes occur.
• Mark an owner responsible for keeping the runbook current and ensuring onboarding includes runbook training.

Templates & Quick Snippets

Quick triage checklist

1. Confirm alert and severity.
2. Identify affected pipelines, tables, and consumers.
3. Check recent commits/deploys and schema changes.
4. Run record-count sanity queries across source and target.
5. Apply containment: pause ingest or switch to cached data.

Backfill pattern (when applicable)

1) Identify earliest corruption point; 2) extract unaffected source window; 3) run transform in controlled environment; 4) validate sample outputs; 5) replay to target; 6) run quality checks and notify consumers.

Where to Tailor this Runbook

Customize the following for each product:

• Concrete metric names and thresholds used by monitoring
• Contact list, escalation matrix, and on-call rotations
• SLA/SLO definitions and measurement source
• Data lineage links and queries for quick verification
• Automated remediation scripts and rollback procedures

Useful Links & Attachments

• Monitoring dashboard: [link]
• Incident tracker template: [link]
• SLA document: [link]
• Schema registry entry: [link]
• Data lineage map: [link]

Change Log

Record edits to this runbook: date, author, summary of changes.