Data Quality & Observability Playbook

Overview

This playbook helps teams detect, classify, and resolve data problems quickly so reports, models, and operational decisions remain trustworthy. It focuses on pragmatic checks you can implement fast, practical observability signals, sensible alerting patterns, clear triage and SLA rules, and concrete remediation runbooks (including automated reprocessing patterns) that close the loop with ownership and post‑mortem learning.

Who should use this

Data engineers, analysts, ML engineers, platform owners, SREs for data, and product owners who rely on data-driven decisions. Use it as a baseline you adapt to domain rules, technologies, and risk profiles.

How to use this playbook

1. Adopt a small set of checks tied to real decisions (start small).
2. Define detection and resolution SLAs per signal severity.
3. Implement lightweight observability signals (metrics + lineage + sample records).
4. Run triage using the runbooks below; log incidents and close with RCA and preventive actions.

Core data quality dimensions

• Schema validity — field types, missing fields, incompatible changes.
• Completeness/volume — counts, expected partitions, missing time windows.
• Distributional consistency — shifts in aggregates or percentiles vs baseline.
• Referential integrity — foreign key violations, orphaned records.
• Freshness/timeliness — latency of ingestion and pipeline stages.
• Accuracy/business rule validity — domain constraints, ranges, allowed values.
• Uniqueness/cardinality — duplicates where uniqueness is required.
• Lineage & observability — traceability from source to consumer.

Practical checks (examples)

Each check should include: what is measured, baseline or expectation, detection threshold, owner, and downstream impact.

Schema check

Detect unexpected column additions/removals or type changes.

Volume / completeness

Distributional drift

Referential integrity

<code>SELECT COUNT(*) FROM events e LEFT JOIN dim_users u ON e.user_id = u.id WHERE u.id IS NULL;</code>

Freshness

Business rule checks

Observability architecture patterns

• Signal types: metrics (counts, lag, error rates), lightweight logs (error class + sample row id), lineage metadata, and periodic sample records (data snapshots).
• Centralized store: pipeline metrics and incidents flow to a central monitoring system or data quality dashboard for correlation and owner routing.
• Data contracts & schema registry: publish expected schema and basic docstrings; consumers subscribe to change notifications.
• Sampling: capture minimal record samples on anomalies to speed triage without storing full payloads.

Alerting & triage flow

Design alerts for action, not information. Each alert must include a suggested owner and a next action.

1. Detection: automated check fires with context (metric, sample, lineage).
2. Classification: type (schema, volume, distribution, referential, freshness, business rule) and severity (P0, P1, P2).
3. Owner routing: map signal type to team/role (ingestion, transformation, model owner, downstream consumer).
4. Triage: owner assesses impact — can consumer tolerate while fix in progress? Start remediation runbook.
5. Resolution: apply fix, validate via checks, document RCA, and close incident.

Example SLA matrix

Define detection and resolution SLAs by severity (tailor to your context):

• P0 (Critical) — data causing incorrect billing, regulatory risk, or model break: Detect < 15 min, Response < 30 min, Resolution < 4 hours.
• P1 (High) — major report divergence, model degradation: Detect < 60 min, Response < 2 hours, Resolution < 24 hours.
• P2 (Medium) — localized data errors or minor drift: Detect < 24 hours, Response < 48 hours, Resolution < 7 days.

Remediation runbooks (templates)

Use a short, repeatable runbook per common failure mode. Each runbook: symptoms, immediate containment steps, root cause probes, fix, validation, follow-up tasks.

Runbook: Schema change detected

1. Containment: Stop dependent downstream jobs if schema change breaks consumers (use feature flags or schema compatibility checks).
2. Probe: Identify change origin (source system vs pipeline migration) and impacted consumers via lineage.
3. Fix: If source introduced compatible change, update mapping/adapters. If breaking change, revert deploy or apply adapter in transformation stage.
4. Validate: Run schema check and sample ingestion for several partitions.
5. Follow-up: Publish change note, update contract, schedule review to prevent repetition.

Runbook: Volume drop (ingestion gap)

1. Check upstream pipeline health and source availability.
2. If transient, reprocess missing partitions; if permanent, run backfill with dedupe safeguards.
3. Validate by comparing to previous day/week baselines and consumer sanity checks.

Runbook: Referential integrity failures

1. Identify offending records and time ranges.
2. Decide containment: quarantine bad records into holding table for manual review or apply best-effort enrichment (if safe).
3. Fix source or transformation logic; run targeted reprocess for affected window.

Automated reprocessing patterns

• Idempotent replays: design transforms so repeated processing of the same upstream data has no adverse effect (use upserts with versioning or dedupe keys).
• Checkpointing & partitioned processing: reprocess only affected partitions to limit blast radius.
• Backfill with safety: include uniqueness keys and validation steps to prevent duplication or regression.
• Test harness: run reprocess in staging and run sanity checks before promoting to production.

Ownership & closing the loop

Every alert must end with a named owner, an RCA, and a preventive action tracked as a ticket. Postmortems should capture detection latency, time-to-resolution, root cause, and a measurable action to reduce recurrence (e.g., new check, enhanced contract, improved alert threshold).

Suggested metrics & dashboards

• Mean time to detect (MTTD) and mean time to resolve (MTTR) per severity.
• Number of incidents by type and owner.
• Check health: percentage of checks passing per pipeline.
• Trend graphs for distributional drift metrics.

Quick-start checklist

1. Pick 6–8 pragmatic checks tied to high‑impact consumers (schema, freshness, volume, referential, top business rule checks).
2. Publish owners and SLAs for each check.
3. Implement metrics and lightweight sampling for failed checks.
4. Create alert routing rules and a simple triage flow (who does what first).
5. Enable automated reprocess for one common failure mode and test it.
6. Run a tabletop incident drill using a realistic failure scenario.

Tailor to your environment

Adapt thresholds, SLAs, ownership, and runbooks to your domain risk, user tolerance, and technical architecture. Start small, measure improvement (MTTD/MTTR), and expand checks that prevent recurring incidents.

Resources & next steps

Suggested follow-ups: implement a schema registry, add lineage metadata to critical flows, create a centralized incident dashboard, and experiment with lightweight anomaly detection for distributional drift.