Data Platform Reference Architecture & Runbook

Purpose

This reference provides concrete architecture templates, a responsibilities matrix, recommended service-level targets, operational runbooks for onboarding and incidents, a deployment checklist, and a short decision guide to help platform teams standardize delivery and reduce fragile one-off pipelines. Use these templates as starting points and adapt them to your technology stack, scale, compliance needs, and organizational roles.

Quick architecture templates

Each template describes common layers, recommended responsibilities, and suggested implementation patterns. Diagrams should reflect your selected technologies (e.g., cloud managed services, on-prem clusters, message brokers, object stores). Typical logical layers:

• Ingestion / Edge — APIs, event collectors, connectors, change-data-capture (CDC) agents, edge buffering. Goal: reliable capture with backpressure and replay capability.
• Landing / Raw Store — immutable storage of original payloads (object store or raw topic). Goal: single source of truth for reprocessing.
• Processing / Compute — batch and stream processing (ETL/ELT, enrichment, deduplication, windowing). Goal: reproducible transformations with lineage and idempotence.
• Metadata & Catalog — schema registry, data catalog, lineage, data contracts, and access policies. Goal: discoverability and governance.
• Serving / Consumption — curated datasets, materialized views, marts, APIs, BI models. Goal: low-latency, consumer-tailored access with clear SLAs.

Canonical pattern

Landing Zone → Raw Zone (immutable) → Enriched / Curated Zone → Serving Layer. Attach metadata and lineage at each handoff.

Recommended SLAs (examples to adapt)

Responsibilities matrix (recommended roles)

Use this as the baseline; adapt role names to your organization.

Legend: R = Responsible, A = Accountable, C = Consulted

Onboarding runbook: adding a new data source

1. Intake & agreement: Data owner completes an onboarding form (dataset purpose, sensitivity, expected volume, SLAs, retention). Capture data contract terms.
2. Assess fit: Choose an ingestion pattern (CDC, batch import, push API, event stream) based on latency, throughput, and reliability needs.
3. Provision resources: Create storage locations, topics, schema registry entries, and access controls with templated names and tags.
4. Implement ingestion: Configure connector/agent and map fields to agreed schema. Record transformation points and durable keys.
5. Quality & test: Run smoke tests, end-to-end replay tests, and data quality checks (nulls, ranges, uniqueness). Include test data set and expected checks.
6. Lineage & catalog: Register dataset in the catalog with description, contact, SLA, PII flags, and lineage links to upstream sources.
7. Deploy & monitor: Promote to production, create dashboards and alerts for ingestion rate, lag, error rate, and data quality failures. Schedule a post-launch review.

Pipeline deployment checklist

• Data contract signed and documented in catalog
• Schema registered and versioned
• Idempotent processing and replay capability verified
• Automated tests for transformations and critical quality gates in CI
• Observability: metrics, logs, traces, and lineage links configured
• Alerting thresholds and escalation path defined
• Access controls and encryption verified for data at rest/in transit
• Retention and archival policy specified
• Runbook and run-level owner documented

Decision guide: batch vs streaming (concise)

Consider streaming when you need low-latency results, continuous correctness for time-ordered events, or incremental updates. Choose batch when you can accept periodic updates, want simpler operational semantics, or need large-window reprocessing.

• Latency needs: Streaming if < seconds/minutes required; batch if minutes/hours are acceptable.
• Event-time semantics: Streaming better if event-time ordering, late arrivals, and windowing matter.
• Volume & cost: High sustained volume can favor streaming for efficient incremental processing; low-frequency large bulk may be cheaper as batch.
• Complexity: Streaming adds operational and testing complexity; prefer batch where it solves the same problem with less risk.
• Reprocessing: Batch reprocessing is often simpler; streaming requires design for stateful rewind or changelog-based rebuilds.

Observability & telemetry recommendations

Instrument pipelines with these metric families and store dashboards/alerts as code where possible:

• Throughput (records/sec), bytes/sec
• Processing latency / end-to-end lag
• Error rates and retry counts
• Data quality gate failures (counts + sample rows)
• Storage growth and retention compliance
• Lineage completeness percentage

Capture structured logs and link traces to lineage to speed root cause analysis. Provide a lightweight incident runbook for common failure modes (connector fail, schema drift, backpressure, permission error).

Incident runbook (common steps)

1. Detect & classify (alerts map to incident severity)
2. Contain (pause consumers, isolated replays, or short-circuit bad data)
3. Notify owners (platform + data owner + SRE)
4. Diagnose (logs, metrics, last good offset, schema changes)
5. Remediate (fix connector/config, restore from raw, backfill, or apply correction job)
6. Review & prevent (post-mortem, adjust SLAs, add tests/alerts)

Security, compliance and governance

Embed privacy and compliance checks early: classify data sensitivity on intake, enforce encryption, limit lateral movement with IAM, log access, and attach retention/archival policies. Use the catalog to publish PII flags and retention schedules.

How to use and adapt this reference

These are practical defaults intended to be tailored. Start by copying the responsibilities matrix and SLA table into your team playbook and adapt thresholds based on consumers. Convert the onboarding runbook to an interactive intake form to collect standardized metadata. Treat the deployment checklist as a CI gate before production promotion.

Helpful next steps

• Turn the onboarding runbook into an interactive intake (forms + required fields)
• Create pipeline templates and connector blueprints for common sources
• Automate catalog registration as part of CI/CD
• Define a limited set of standard SLAs and enforce them via dashboards and monthly reviews

Glossary (brief)

CDC = change data capture. Raw zone = immutable storage of ingested payloads. Data contract = agreed schema and semantics between producer and consumer. Lineage = recorded data flow from source to consumption.