Data Instrumentation Plan for Discovery

1. Key events to capture (example taxonomy + schema suggestions)

Capture events that matter to the experiment’s outcome and to reusing the dataset later. For each event, include an intent line, required properties, recommended types, and one example instance.

Event: experiment.session_start

• Intent: mark beginning of an experiment session for a user or device.
• Required properties: session_id (string), user_id (string | null), experiment_id (string), timestamp (ISO8601)
• Recommended: device_type, locale, sampling_rate
• Example (JSON):

{
  "event": "experiment.session_start",
  "session_id": "s_9b3f...",
  "user_id": "u_1245",
  "experiment_id": "exp_checkout_v2",
  "timestamp": "2025-06-01T10:12:00Z",
  "device_type": "mobile"
}

Event: experiment.checkout_action

• Intent: record attempt, success, or failure of a key action.
• Required properties: event_type (string: attempt|success|failure), session_id, user_id, amount (number | optional), timestamp
• Recommended: error_code, latency_ms, feature_flags

Guidance: prefer consistent property names across events (e.g., always use "user_id" and a standard timestamp). Use enumerations for event_type and document allowed values.

2. Responsible owner(s) and data steward

Document clear ownership to avoid fragmented, duplicated, or stale instrumentation.

• Event Owner (product/feature): name, role, contact — accountable for event definition and product semantics.
• Data Steward (analytics/engineering): name, role, contact — responsible for schema enforcement, downstream datasets, and PR reviews.
• Approval Process: how schema changes are proposed, reviewed, and deployed (e.g., pull request + data steward sign-off).

3. Acceptance criteria for telemetry

Define measurable criteria before deployment. Examples:

• Completeness: >= 98% of sessions must include session_start within 10s of user activity (measured in QA environment)
• Accuracy: timestamps within expected clock skew; numeric fields in expected ranges (e.g., amount >= 0)
• Uniqueness/Idempotence: events with the same operation should include a stable event_id to deduplicate retries
• Latency: telemetry ingest latency < 2 minutes for critical experiment metrics (or state expected for your use case)
• Privacy/Compliance: PII must be removed or tag with privacy level and retention policy enforced

4. Quick verification tests and dashboards

Smoke tests to run immediately after deploy (automate when possible):

1. Instrumented Action Test — perform the action and verify the event appears in raw event stream within expected latency.
2. Schema Validation — run a validation query to confirm required fields are non-null and types match.
3. Rate Check — compare event counts to expected traffic sample to detect large drops or spikes.
4. Duplicate Detection — run a dedupe check for repeated event_ids over a short window.
5. Downstream Metric Sanity — check the experiment’s key metric on the dashboard matches manual calculation for a small sample.

Suggested lightweight dashboard panels:

• Event volume (last 24h, 7d) for each critical event
• Latency histogram for ingestion and processing
• Acceptance criteria KPIs (completeness %, schema errors/day)
• Error-rate panel showing events with error_code or failure event_type

5. Rollout and monitoring notes

• Sampling: note any sampling decisions and how to reconstruct unsampled estimates.
• Retention and export: retention policy for raw events and which datasets are materialized for discovery.
• Privacy tags: label fields with privacy level (public, internal, restricted, pii) and required handling.
• Failure modes: what to do if events stop arriving (alerting runbook), including key contacts.

6. Common pitfalls & trade-offs

• Over-instrumentation: capturing every field increases cost and maintenance. Start with required fields and an explicit "opt-in" for extra properties.
• Inconsistent names: avoid synonyms (e.g., "uid" vs "user_id"). Use a short style guide for property names and enumerations.
• Heavy-handed governance: require lightweight change control for experiments; keep a path to promote stable events to enterprise datasets with stronger governance.
• Shadow datasets: track derived datasets and owners to prevent fragmentation and duplicate cleaning efforts.

Template Checklist (copy into your experiment card)

• Key events listed with required properties
• Owner and data steward named
• Acceptance criteria defined
• Verification tests written and automated where possible
• Dashboards or monitors created
• Privacy and retention rules recorded
• Rollback and failure contacts included

Use this checklist as part of your deploy or experiment kickoff workflow to reduce delays in drawing conclusions and to preserve reusable, trustworthy datasets.