Data Observability Runbook & Alerting Recipes

Purpose and scope

This runbook helps teams detect, triage, and resolve common data quality incidents so reports, models, and operational systems remain trustworthy. It focuses on practical checks, alerting recipes, incident triage, ownership, and reducing alert fatigue. Use it for pipelines, datasets, and feature stores that feed analytics, dashboards, and ML models.

Quick navigation

• Classification of anomalies
• Detection recipes (with example checks)
• Triage playbook
• Common fixes and automation
• Communication templates
• Alert prioritization & noise reduction
• Vendor-selection checklist
• Metrics to measure observability

Classification of anomalies

Classify each alert to speed triage and route ownership:

• Schema — unexpected column add/remove, type change, missing required fields.
• Volume — sudden drops or spikes in row counts or file sizes.
• Freshness — late or missing runs, data lag beyond SLA.
• Distribution / Drift — changes in value distributions, percentiles, or feature drift for ML inputs.
• Completeness / Null rates — new nulls, missing partitions, or unexpected defaults.
• Uniqueness / Integrity — duplicate primary keys, foreign key violations.
• Business rule violations — e.g., negative prices, impossible dates.

Detection recipes

For each dataset or pipeline, implement a small set of prioritized checks tied to real decisions. Examples below assume a SQL-capable warehouse.

Schema checks

Compare observed columns to a canonical schema. Alert if required columns are missing or types differ.

Example (pseudo-SQL): SELECT column_name, data_type FROM information_schema WHERE table='my_table' -- compare to expected list

Freshness check

Alert when the latest ingestion time is older than SLA (e.g., 4 hours):

SELECT MAX(ingest_time) AS last_time FROM my_table; -- alert if last_time < now() - interval '4 hours'

Volume check

Compare daily row count to rolling median. Alert on large % deviation (example: >30% drop):

WITH counts AS (SELECT date_trunc('day', created_at) d, count(*) c FROM my_table GROUP BY 1)
SELECT c, median_7d, (c - median_7d) / NULLIF(median_7d,0) pct_change FROM (select d,c,percentile_cont(0.5) within group (order by c) OVER (ORDER BY d ROWS BETWEEN 7 PRECEDING AND 1 PRECEDING) as median_7d from counts) t WHERE d = current_date;

Distribution / drift

Monitor key percentiles, mean, stddev, and use a simple statistical test (KS test or KL divergence) to flag drift. For categorical data, watch top-category share changes.

Null-rate and cardinality checks

SELECT count(*) AS total, sum(CASE WHEN important_col IS NULL THEN 1 ELSE 0 END) AS nulls, (nulls*1.0/total) null_rate FROM my_table;

Alert when null_rate > threshold (e.g., 10%) or cardinality drops below expected.

Anomaly scoring & thresholds

Prefer relative thresholds tied to historical variability (z-score, percentiles) rather than arbitrary fixed numbers. Flag anomalies when score > 3 sigma or when percentage change exceeds business-defined impact thresholds.

Triage playbook (step-by-step)

1. Confirm — verify the alert is valid (not a transient false positive). Check raw source, recent job logs, and run history.
2. Classify — label with anomaly type (schema, freshness, volume, etc.) and estimate potential impact (reporting, SLAs, ML models, customers).
3. Scope — determine affected datasets, downstream consumers, and time range. Is the issue isolated or systemic?
4. Assign owner — route incident to the appropriate team (ingestion, transformation, platform, data product owner). Record ownership in ticket/incident system.
5. Remediate — choose path: automated retry/backfill, manual data fix, upstream coordination, or rollback. If automated remediation exists, run it and validate.
6. Communicate — send an incident status note to stakeholders (see templates below). Include runbook link, owner, estimated ETA, and impact.
7. Validate & close — confirm data is repaired, downstream dashboards/calculations are correct, and update incident ticket with root cause and actions taken.
8. Postmortem — for high-impact or recurring incidents, run a short blameless postmortem. Record permanent fixes (e.g., contract, additional checks, automation).

Common fixes and automation patterns

• Automatic pipeline retries for transient infra failures.
• Backfill jobs to replay failed partitions or missing dates.
• Quarantine bad rows into a staging table for manual review.
• Automatic schema migration guardrails: treat breaking changes as pull-requests and require owner signoff.
• Auto-rollback of schema changes for consumer-impacting migrations.
• Small remediation scripts (re-parsing, reformatting, casting types).

Communication templates

Initial alert (short)

[ALERT] [dataset] [impact]
Owner: @team
What: brief description (e.g., row count down 60%)
Where: dataset/table name, partition/date
When: detected at T
Action: triage started (link to runbook & ticket)
ETA: initial estimate

Status update (example)

We identified a schema mismatch in orders_v2. Owner: @data-platform. Cause: upstream change to event producer. Status: rollback applied and backfill started. Next update: in 30 minutes.

Postmortem summary

Describe timeline, root cause, impact, actions taken, permanent fixes, and owners for each fix (including monitoring to add).

Alert prioritization — reduce noise

• Only alert when action is required. Use warning-level logs for non-actionable anomalies.
• Map alerts to business impact (P1: reporting/ops broken, P2: reduced model accuracy, P3: informational).
• Group related alerts into a single incident (e.g., many partitions failing from one upstream job).
• Use deduplication and rate limiting on noisy checks. Suppress repeated alerts until state changes.
• Attach ownership and runbook links to every alert so responders know next steps immediately.
• Score alerts by confidence and recent false-positive history; route high-confidence alerts to paging channels.

Vendor / tooling selection checklist

• Supported checks: schema, freshness, volume, distribution, nulls, uniqueness.
• Integrations: alerting (PagerDuty/Slack/Email), orchestration (Airflow, dbt), and warehouses.
• Lineage & root-cause: ability to trace upstream jobs and owners.
• Noise controls: dedupe, grouping, suppressions, confidence scoring.
• Automation & remediation hooks (APIs to trigger retries/backfills).
• Scalability & cost for your data volume and frequency.
• Security, compliance, and access controls.

Key observability metrics

• Mean time to detect (MTTD) — time from fault to alert.
• Mean time to remediate (MTTR) — time from detection to verified fix.
• Alert volume — total alerts per week (and trend).
• Signal-to-noise ratio — percent of alerts that are actionable/real.
• Percent of alerts with an assigned owner within 15 minutes.
• Recurring incidents — number of repeat faults by dataset (target: downward trend).

How to use this runbook

Copy the detection recipes into your monitoring tool or SQL-based checks. Start small: implement a handful of checks for critical datasets tied to real business decisions. Add automation for repeatable remediations. Review noisy alerts weekly and refine thresholds or add grouping. Periodically run a tabletop drill to validate triage flows and communication templates.

Link this runbook from every alert generated so responders land directly on next steps, owner contacts, and remediation scripts.

Appendix — example threshold guidance

• Freshness: alert if lag > SLA or 2x historical typical lag.
• Volume: alert if current < 70% of rolling 7-day median or > 300% spike (tune by dataset volatility).
• Null rate: alert if increase > 10 percentage points or > 3x baseline.
• Distribution drift: KS p-value < 0.01 or KL divergence > tuned threshold.