Experiment Runbook: Design, Execute, Analyze

Welcome — Run experiments that lead to reliable decisions

This runbook gives a concise, operational process your team can follow every time you run an experiment. Use it to reduce noise, avoid common biases, protect customers and systems, and convert learning into prioritized, measurable decisions. The runbook is organized so a busy team can pre-register intent, confirm instrumentation, launch safely, analyze rigorously, and capture practical lessons.

How to use this runbook

Copy the templates into your project doc, fill the brief before you build, use the checklists before launch, and follow the analysis & interpretation rules exactly as pre-registered. Treat the post-mortem as the source of truth for next steps.

Quick-start checklist (one-page)

1. Write a short experiment brief (hypothesis, audience, primary metric, success threshold).
2. Confirm sample & power plan (minimum sample, test length, stopping rules).
3. Validate instrumentation & telemetry (events, tags, dashboards).
4. Pre-register analysis plan (primary/secondary metrics, tests, covariates).
5. Set monitoring & safety thresholds and on-call contacts.
6. Launch controlled segment; monitor in real time for safety signals.
7. Lock analysis decisions at pre-registration end; run analysis and interpret using decision rules.
8. Run post-mortem and convert learning into decisions and backlog items.

Experiment brief (template)

Keep this to one paragraph where possible.

• Title: (Short descriptive name)
• Owner(s): (Who runs & who decides)
• Start Date / Planned End Date:
• Audience / Segment: (Who is eligible)
• Hypothesis: "If we change, then metric will direction by amount for audience because rationale."
• Primary metric & unit: (exact event or KPI, e.g., daily active purchases per user)
• Success threshold: (practical minimum improvement or harm threshold; vetted with stakeholders)
• Safety / guardrail metrics: (metrics you will watch to stop or rollback)
• Scope & rollout plan: (percent traffic, segmentation, phasing)

Sample & power planning checklist

Goal: choose sample size or test duration that produces useful evidence while limiting exposure.

1. Define the primary metric precisely (numerator, denominator, aggregation window).
2. Estimate baseline mean/rate and variance from recent historical data.
3. Decide minimum detectable effect (MDE) that would change decisions.
4. Choose desired power (commonly 80%); choose alpha (commonly 0.05 two-tailed).
5. Calculate sample size or test duration using a standard calculator. Document assumptions.
6. If traffic is limited, consider increasing test duration, using hierarchical models, or switching to a paired or within-subject design where valid.
7. Define explicit stopping rules (fixed-horizon preferred; avoid peeking unless sequential methods are pre-registered).

Note: When in doubt, choose an MDE tied to business value (e.g., change must generate $X over Y months) rather than an arbitrary percent.

Instrumentation & telemetry checklist

Before you launch, confirm:

• Primary event(s) are instrumented and named exactly as in the analysis plan.
• Variant identifiers and assignment logs are captured (who saw A vs B, timestamp, user id where privacy allows).
• Client/server-side event parity is confirmed (no significant loss on any platform).
• Key dimensions for segmentation (country, platform, cohort, new vs returning) are available.
• Dashboards exist for: primary metric over time, sample counts, variant balance, and guardrail metrics.
• Alerting is configured for severe negative impacts (error rates, revenue drops, latency spikes).
• Data retention and privacy constraints are documented and respected.

Launch steps

1. Deploy to a small pilot (e.g., 1–5%) if feasible to validate instrumentation in production.
2. Run pilot for a short smoke period (24–72 hours) and confirm metrics and events look as expected.
3. If pilot is healthy, scale to full planned allocation.
4. Log launch time, exact variant splits, and any runtime anomalies.

Monitoring & safety thresholds

Define clear automatic and manual actions.

• Automatic rollback thresholds: e.g., >5% increase in error rate sustained for 30 mins, or >10% drop in revenue metric in an hour. If triggered, roll back and investigate.
• Manual review thresholds: moderate signals that require on-call review before acting.
• Assign on-call person(s) and communication channels (Slack, pager, email) for escalation.
• Document expected transient noise and what counts as an actionable signal.

Analysis plan & pre-registration prompts

Lock the following before viewing final results:

• Primary hypothesis and metric: restate exactly from the brief.
• Primary statistical test or model: e.g., two-sample t-test on user-level conversion, Poisson regression for counts, or a hierarchical Bayesian model.
• Significance level and directionality: alpha, one/two-tailed.
• Pre-specified covariates or adjustments: baseline, platform, cohort.
• Missing data handling: exclude, impute, or intention-to-treat; be explicit.
• Multiple comparisons plan: whether you will correct for multiple tests and how.
• Analysis window: which dates and why (e.g., avoid seasonality).
• Decision rules: define exactly what counts as "deploy", "iterate", or "rollback" based on evidence and business impact.

Interpretation guide

Use these rules to translate numbers into decisions:

• Clear positive: Primary metric shows statistically significant improvement beyond the success threshold and no guardrail violations → consider rollout and productization with a prioritized implementation plan.
• Clear negative: Statistically significant harm or guardrail breach → stop and roll back. Run a diagnostics post-mortem.
• Null or inconclusive: Effect size below MDE or confidence intervals include both meaningful harm and benefit → don't assume success. Options: increase sample, refine the intervention, or deprioritize based on cost/benefit.
• Heterogeneous effects: If subgroups vary widely, investigate mechanism before large rollout; consider targeted rollouts to segments that benefit.

Post-mortem template

Complete this within five working days of analysis.

• Experiment brief (link):
• Outcome: primary metric result, confidence interval, p-value, decision (deploy/iterate/stop)
• Secondary & guardrail findings:
• What worked / surprises:
• Root cause analysis (if negative):
• Next steps (owners & due dates): full rollout, additional experiments, instrumentation fixes, or process changes
• Knowledge artifacts to save: datasets, scripts, dashboards, and a short plain-language summary for stakeholders

Short worked example

Title: "Simplified checkout flow" — Hypothesis: Reducing steps in checkout will increase conversions by at least 3% (primary metric: conversion rate per checkout session). Baseline: 12% conversion rate. MDE: 3% absolute (to 15%). Power calc -> N = X per variant (documented). Launch to 20% traffic after a 48-hour pilot. Pre-registered t-test at alpha=0.05, power=80%. Monitoring: payment error rate and revenue per user as guardrails. Result: +1.2% (CI [-0.5, 2.9]) -> inconclusive; decision: iterate (refine UI and run follow-up targeted to mobile users where variance was highest).

Links & tools

• Sample size & power calculators: common online calculators (search "two-sample proportion power calculator")
• Dashboards: link to your analytics workspace and instrumentation runbook
• Statistical methods reference: brief internal guide or a trusted public reference (e.g., cookbook on A/B testing best practices)

Final notes

Preserve the pre-registered plan and record any deviations with rationale. Treat experiments as managed learning investments: protect participants, prefer clarity over speed when evidence is weak, and translate every result into prioritized actions.