Experimentation Protocol & Runbook Template

Purpose

This template captures a complete, pre‑registered experiment protocol and runbook so teams can design rigorous tests, reduce bias, ensure reproducibility, and turn results into clear decisions. Use it as a copy/paste starting point for A/B tests, pilot interventions, operational trials, and controlled experiments.

How to use

Fill every field before you launch. Pre‑register the Analysis Plan and Stopping Rules. Keep instrumentation and analysis artifacts (SQL / code / export) with the experiment record so others can reproduce results. When possible, link to the code repo, schema, and dashboards.

Experiment Metadata

• Experiment ID: ______________________
• Title: ______________________
• Owner / Sponsor: ______________________
• Team / Stakeholders: ______________________
• Date proposed: ______________________
• Status: Draft / Pre‑registered / Running / Analysis / Complete

1. Hypothesis & Primary Outcome Metric

Write a crisp, testable hypothesis and name a single primary outcome metric that will determine success.

Hypothesis (If X, then Y): ______________________

Primary outcome metric (definition and units): ______________________

Why this metric matters: ______________________

Secondary metrics (safety, leading indicators, guardrails):

• Secondary 1: ______________________
• Secondary 2: ______________________

2. Randomization & Sampling Plan

Define the population, sampling frame, and how assignment will be randomized.

• Population / eligibility criteria: ______________________
• Exclusions / opt‑outs: ______________________
• Unit of randomization (user, session, store, device): ______________________
• Assignment method (e.g., hashed user ID, stratified block rand): ______________________
• Stratification / blocking variables (if any): ______________________
• Rollout cohorts / waves (if phased): ______________________

3. Data Collection & Instrumentation Checklist

Ensure reliable measurement before launch. Confirm each item.

• Event names and schema documented and agreed (list events): ______________________
• Primary metric derivation SQL / code location: ______________________
• Experiment assignment logged for every unit (assignment log location): ______________________
• Logging for secondary and guardrail metrics enabled
• Data retention, privacy, and consent requirements verified
• Smoke tests and pre‑launch checks completed on a staging dataset
• Monitoring dashboard(s) set up for live checks (link): ______________________

4. Power & Sample‑Size Notes

Record assumptions used to compute sample size or power. If running an exploratory or qualitative pilot, state that explicitly.

• Baseline rate / mean of primary metric: ______________________
• Minimum Detectable Effect (MDE) / effect size deemed meaningful: ______________________
• Alpha (type I error): 0.05 (or specify)
• Desired power (1−beta): 0.8 (or specify)
• Calculated sample size per arm: ______________________
• Estimated duration to reach sample (traffic & conversion assumptions): ______________________

Helpful calculator: https://www.evanmiller.org/ab-testing/sample-size.html (replace with your org’s approved tool if available).

5. Acceptance, Stopping Rules & Rollout Plan

Specify precise decision criteria before seeing the results.

• Primary decision rule:

  e.g., "If the primary metric shows a statistically significant improvement at alpha=0.05 and the effect ≥ MDE, accept and proceed to ramp."

• Futility / harm rules:

  e.g., "Stop early if the 95% CI is entirely below 0 improvement or if guardrail metrics degrade by > X% for 3 consecutive days."

• Interim looks and multiplicity:

  Specify whether interim analyses are allowed and how you'll adjust for multiple looks (alpha spending, Bonferroni, or pre‑specified fixed looks).

• Rollout plan if positive:
  1. Validate on a holdout / different population
  2. Ramp to X% over Y days while monitoring guardrails
  3. Full rollout with post‑launch verification

6. Analysis Plan & Pre‑Registered Report Stub

Describe exactly how you will analyze data. Include model specs, exclusion rules, and what constitutes the analysis population.

• Primary analysis approach: (two‑sample t, proportion test, regression, difference‑in‑differences, etc.) ______________________
• Covariates / adjustments (if any): ______________________
• Handling of missing data / attrition: ______________________
• Outlier rules / winsorization: ______________________
• Subgroup analyses (pre‑specified): ______________________
• Multiple comparisons correction (if relevant): ______________________
• Primary result to report: point estimate, 95% CI, p‑value, sample sizes
• Analysis code location / reproducibility artifacts: repo / notebook / script link

Pre‑registered report (copy‑paste stub):

We will test the hypothesis that [brief hypothesis]. The primary outcome is [metric definition]. We will randomize [unit] using [method]. The sample size per arm is [n], calculated assuming baseline [value], MDE [value], alpha [value], power [value]. The primary analysis will use [stat test/model]. Missing data will be handled by [method]. We will declare the experiment "positive" if [decision rule]. Analysis code is stored at [link].

7. Decision Rubric (Action Matrix)

Predefine actions to avoid ad‑hoc post hoc decisions.

• Positive (meets acceptance rule): Validate → Ramp → Full rollout; create task to update docs and train ops.
• Inconclusive (no clear signal, underpowered): Consider extended run, increase sample, or redesign with clearer primary metric.
• Negative (harm or no effect): Stop and document learnings; consider alternate designs if the hypothesis remains important.

8. Post‑Experiment Artifacts & Learning

• Final analysis report (link): ______________________
• Key takeaways and next steps: ______________________
• Retrospective: what surprised us, what to change in future experiments
• Archive dataset and code path: ______________________

Quick Launch Checklist

1. All metadata fields filled and pre‑registered.
2. Instrumentation verified on sample data.
3. Primary and secondary metrics computed and reproducible.
4. Power/sample assumptions documented and realistic.
5. Acceptance & stopping rules agreed by stakeholders.
6. Monitoring dashboards ready and alerted.
7. Communication plan for rollout and rollback prepared.

Example (short)

Hypothesis: Adding a one‑click checkout button will increase checkout conversion by at least 3 percentage points. Primary metric: checkout conversion rate (orders / sessions). Unit: user session. Sample size per arm: 10,000 sessions (based on baseline conversion 8%, MDE 3pp, alpha 0.05, power 0.8). Assignment: hashed user ID mod 100. Acceptance rule: statistically significant increase in conversion ≥ 3pp at alpha 0.05 and no deterioration in average order value or payment failure rate.

Designed for copy‑paste

Use the fields above as a standardized runbook in your experimentation platform. Keep links to code, dashboards, and datasets with the experiment record. If your organization supports an experiment registry or interactive form, consider capturing this template as structured fields so experiments are searchable, comparable, and auditable.