Toolbox

Experimentation Protocols & A/B Templates

Interactive experiment protocol and A/B template: hypothesis framing, primary-metric definitions, sample-size guidance, logging & instrumentation checklist, guardrail metrics, analysis plan, and result-interpretation checklist. Saves protocols for reproducibility and handoff.

Interactive Tool

Experimentation Protocol & A/B Template

This interactive protocol helps teams design, document, and save rigorous A/B and pilot experiments. Use it to lock down your hypothesis, define primary and guardrail metrics, capture sample-size assumptions, confirm instrumentation, and record your planned analysis so results are interpretable and actionable.

Quick guidance on sample size: capture your baseline rate and the minimum detectable effect (MDE) you care about, pick a significance level (α) and power (1−β). A commonly used reference calculator is Evan Miller's sample size tool; if you prefer a simple approximation for proportions, paste the calculator output into the sample-size note below. Always plan for potential data loss and check statistical assumptions before acting.

When you save this form you create a persistent experiment protocol that helps prevent common mistakes: underpowered tests, undefined metrics, missing instrumentation checks, and ambiguous interpretation. Fill the form before launching and update it with results after the test concludes.

Short, descriptive name for the experiment (team + goal + date helps).
Person or team responsible for the experiment and decisions.
State intervention and expected change. Example: 'If we simplify checkout, then conversion rate will increase by at least 1.5 percentage points.'
The single metric you'll use for your primary decision (e.g., 'checkout conversion').
Be explicit: numerator, denominator, event names, segmentation, and any inclusion/exclusion filters. This prevents ambiguity during analysis.
Enter baseline as a decimal (e.g., 0.12 for 12%) or percent (12). Use the same unit when entering MDE.
Smallest absolute change you consider practically important (e.g., 1.5 for +1.5 percentage points).
Example: 1:1, 2:1. Affects per-group sample sizes.
Paste output from your sample-size calculator here or record the approximate n per variant. Helpful calculators: Evan Miller's A/B test sample size calculator. Note any assumptions (expected variance, continuity correction, sequential testing).
Specify how users are assigned to variants.
List users, events, or conditions to exclude (e.g., internal traffic, bots, repeat sessions).
Planned duration for the experiment. Consider business cycles and sample velocity.
Run these checks on a dry run or early sample to catch issues.
List metrics you will monitor for adverse side effects (e.g., revenue per user, page load errors, support rate).
Specify statistical tests, adjustments for multiple comparisons, covariate adjustments or regression, handling of outliers, and any pre-registered subgroup analyses.
Include links to analytics dashboards, experiment logs, or tickets for auditability.
You can explore this tool now. Sign in or create an account to save your responses and return to them later.
Sign In to Save Create an Account
Make this tool part of your work

Save a personal copy, bring it to your team, or tailor the questions and workflow to fit what you are hungry to improve.

Member customization and team collaboration are coming soon.
Discuss Ask the Librarian

Discussion

Comments and conversation will live here.