Data Product Runbook & SLA Template

Purpose and Scope

This runbook helps teams operate, stabilize, and evolve a data product. It is a living template intended to be copied and tailored to your environment, monitoring stack, consumers, and governance. Use it to define ownership, measurable SLAs/SLOs, health checks, monitoring, incident playbooks, deployment controls, and communication patterns.

Product Description & Consumers

Include a concise description the team and others can quickly read during an incident:

• Product name: (e.g., Customer Orders Stream)
• Primary owner: Name / Pager / Role
• Consumers: Teams, dashboards, downstream jobs, SLAs that rely on this product
• Purpose: What decision, automation, or report this product supports
• Key inputs/outputs: Upstream sources, schemas, destination tables/feeds

SLAs, SLOs and Measurement

Define what you promise to consumers and how it is measured. Keep SLAs simple, measurable, and aligned to consumer needs.

• Example SLA (availability): 99.5% successful hourly job runs measured over a 30-day window. Penalty/notifications described in governance.
• Example SLOs (service-level objectives):
  • Freshness SLO: 95% of hourly partitions available within 10 minutes of hour boundary.
  • Schema stability SLO: No unapproved schema changes in production over a 90-day period.
  • Completeness SLO: Daily ingestion contains >= 99% of expected records (with threshold defined per source).
• SLI examples (how to measure):
  • Job success rate = successful_runs / total_runs over the measurement window.
  • Max processing lag = max(event_time_lag) across the last hour.

Measurement notes: Record the query, source of truth, and measurement frequency. Include retention and where historical SLI values are stored.

Health Metrics & Thresholds

List a small set (3–8) of actionable health metrics with thresholds that trigger alerts and the intended response level.

• Job success rate (15m): Alert if < 95% in last 15 minutes.
• Processing lag: Alert if max lag > 30 minutes for hourly partition.
• Row volume change: Alert on > 30% deviation from expected daily volume.
• Schema change detected: Alert on any unapproved schema drift.
• Data quality: Alert on failed QA checks (nulls in key columns, referential integrity failures).

Example monitoring queries (adapt to your tooling):

Monitoring & Alerting Integration

For each health metric list:

1. Alert name and severity level (P1, P2, P3).
2. Exact alerting rule (copy the query and thresholds).
3. Notification channels (PagerDuty, Slack #alerts-data-product, email).
4. Runbook link (this document) and on-call roster.

Common Incident Playbooks

Each incident type should have clear triage steps, mitigation, and recovery actions. Keep them short and testable.

1. Job Failure (P1/P2)

1. Confirm failure: check job logs, error messages, recent deploys.
2. Assess impact: which partitions, downstream consumers affected, SLA risk?
3. Triage steps:
   1. Restart job if transient error and restart safe.
   2. If resource exhaustion, escalate to infra/ops and consider temporary throttling.
   3. If code error, identify offending change, revert or patch after review.
4. Mitigation: if backlog grows, start a controlled backfill (see Backfill Playbook).
5. Communicate to consumers using the communication template.
6. Create an incident record and schedule a post-incident review.

2. Upstream Outage

1. Identify which upstream source is degraded. Check provider status pages.
2. Assess downstream impact and whether to pause downstream consumers or mark data as stale.
3. Mitigation:
   • Fallback to alternate source if available.
   • Serve last-known-good snapshot and annotate freshness in downstream reports.
4. Notify stakeholders and update incident record.

3. Schema Change Detected

1. Validate change: was it planned and approved? Check change requests and recent deploys.
2. If unplanned, block downstream consumers from consuming the altered field (feature flag or view).
3. Mitigation: coordinate with source owners to revert or publish a compatible schema migration plan.
4. Run regression tests and roll out migration with canary if needed.

4. Data Quality / Drift

1. Run automated DQ checks to localize the problem (nulls, ranges, distribution shifts).
2. Decide to quarantine affected partitions or tag rows with provenance and suspect flag.
3. Plan corrective backfill if needed and communicate clearly to consumers.

Backfill Playbook (when a controlled reprocessing is required)

1. Scope: define partitions and time window to reprocess.
2. Estimate resources and expected run time. Run in staging if possible.
3. Create a backfill job with throttling to avoid overloading downstream systems.
4. Verify results with data quality checks before marking backfill complete.
5. Update changelog and notify consumers.

Deployment Checklist

Use this checklist for every production deployment. Include it in the PR or release notes.

• Code reviewed and signed off by owner.
• Unit and integration tests passing.
• Schema changes documented and backward compatible or coordinated with consumers.
• Canary deployed and monitored for at least one full processing cycle.
• Alerting rules validated; run health checks after deploy.
• Rollback plan documented and tested.

Versioning & Changelog

Keep a human-readable changelog with every release. Include:

• Release version and date.
• Owner and approver.
• Summary of change and impact (schema, behavior, contracts).
• Migration or backfill steps.

Communication Templates

Copy these templates into your incident channel and customize details before sending.

Initial Incident Notification (to internal stakeholders)

Subject: [P1] {Product} degraded — initial status

Body: We detected {brief issue}. Impact: {consumers/reports affected}. Mitigation: {actions in progress}. Owner: {name}. ETA next update: {time}.

Consumer Notification (external / downstream teams)

Subject: {Product} data freshness impacted (start time — estimated recovery)

Body: We detected {brief issue}. Affected datasets: {list}. Workaround: {if any}. Expected resolution: {ETA}. We will provide updates at {interval}.

Escalation Matrix & Roles

• Owner: Responsible for triage and communications.
• On-call engineer: Executes technical mitigation.
• Platform/Infra: Escalate for resource, network, or storage issues.
• Product manager/business lead: Notified if downstream SLAs or customers are impacted.
• Legal/Compliance: Escalate for data breaches, PII issues, or regulatory concerns.

List exact contact details and back-up contacts here.

Post-Incident Review (PIR) Template

1. Incident summary: timeline and impact.
2. Root cause analysis: what happened and why.
3. Immediate mitigations taken and their effectiveness.
4. Action items: owners, due dates, verification steps.
5. Learnings and changes to runbook, tests, or monitoring.

Store PIRs with a cross-reference to changelog entries, retrospective notes, and updated tests/alerts.

Testing, Exercises & Governance

Runbooks work only if practiced. Schedule quarterly fire drills that exercise backfills, schema-change responses, and production deploy rollbacks. Periodically review SLAs with consumers and update SLOs where business needs change.

Tailoring & Copy Instructions

This template is intentionally generic. When copying for your product, explicitly fill these fields:

• Owner contact details and on-call schedule
• Exact monitoring queries and alert channel IDs
• SLAs/SLOs with measurement query and storage location
• Deployment and rollback playbooks specific to your CI/CD

Appendix: Example Quick Checks

Use these quick checks during triage to decide if an incident requires escalation:

• Are error rates increasing only for one host or across all workers?
• Is there a recent deploy coincident with the start time?
• Are upstream provider health pages reporting incidents?
• Do data volume and schema heuristics indicate drift?

Where to Store This Runbook

Keep the canonical copy linked from the product README, the team on-call runbook index, and the change management system so changes are discoverable and versioned.