Experiment Design Worksheet & Power Calculator

Welcome — what this worksheet helps you do

This worksheet helps teams turn a vague idea into a defensible experiment: clarify the objective, pick primary and secondary metrics, specify the minimum detectable effect (MDE) that matters, calculate required sample size (or achievable power), pre-register analysis and stopping rules, check common risks and biases, and plan a safe rollout. Use it to avoid underpowered tests, ambiguous outcomes, and accidental operational harm.

How to use this tool

1. Fill in the objective and hypothesis clearly.
2. Choose a single primary metric (what success means).
3. Estimate baseline performance and variability from recent data.
4. Decide the smallest effect you care about (MDE) and acceptable error rates (alpha, power).
5. Use the formulas or paste the spreadsheet formulas below to compute sample sizes or power.
6. Write stopping rules, guardrail metrics, and an operational rollout plan before you start.

Worksheet sections

1) Objective & hypothesis

Objective (one sentence):

Example: Increase signup rate for the new onboarding flow to reduce time-to-value.

Primary hypothesis (if-then):

Example: If we show the simplified signup screen, then the proportion of users who complete signup within 5 minutes will increase.

2) Primary metric (single)

Pick one primary metric that best captures the objective. Make it concrete and measurable (e.g., conversion rate, mean time to completion, average revenue per user). Describe calculation and units, and the observation window.

3) Secondary & guardrail metrics

• Secondary metrics: metrics that help interpret the primary result (e.g., click-through, engagement).
• Guardrail metrics: safety, quality, cost, or compliance metrics that must not degrade (e.g., error rate, latency, customer complaints).

4) Assignment & sampling

Population: define who is eligible and how long the experiment will run (calendar vs. sample size).

Assignment method: simple randomization, stratified randomization (recommended for important covariates), cluster randomization (specify clusters), or stepped rollout. If cluster, estimate ICC and average cluster size for design effect.

5) Minimum Detectable Effect (MDE)

State the smallest effect (absolute or relative) you care about that would change your decision. Resist using a very small MDE just because it is technically detectable — think about commercial or operational relevance.

6) Statistical settings

• Alpha (type I error, two-sided default): 0.05
• Power (1 - type II error, typical default): 0.8 (80%) or 0.9 for higher confidence
• Allocation ratio (treatment:control): 1:1 default; adjust if you need more observations in one arm.

Power & sample-size formulas (practical)

Binary outcome (two-sample proportions)

Parameters: baseline proportion p1, treatment proportion p2 = p1 + Δ (Δ is absolute MDE), alpha, power.

Approximate per-group sample size:

n ≈ [ (z_{1-α/2} * sqrt(2 * p̄ * (1 - p̄)) + z_{power} * sqrt(p1*(1-p1) + p2*(1-p2)) )^2 ] / (p2 - p1)^2

where p̄ = (p1 + p2) / 2, z_x is the standard normal quantile (e.g., z_{0.975}=1.96, z_{0.8}=0.84).

Continuous outcome (two-sample means, equal variance)

Parameters: standard deviation σ, absolute difference Δ you want to detect, alpha, power.

Per-group sample size:

n = 2 * ( (z_{1-α/2} + z_{power})^2 * σ^2 ) / Δ^2

Design effect for clustering

If using clusters, multiply sample size by the design effect DE = 1 + (m - 1) * ICC, where m is average cluster size and ICC is the intraclass correlation.

Worked examples

Example A — conversion (binary)

Baseline p1 = 10% (0.10). Target p2 = 12% (absolute Δ = 0.02). α = 0.05, power = 0.80.

Compute z values: z_{0.975}=1.96, z_{0.8}=0.84. Using the formula above gives ~3,828 users per arm (≈7,656 total).

Interpretation: to reliably detect a 2 percentage-point lift from 10% → 12% you need thousands per arm; smaller effects require much larger samples.

Example B — mean (continuous)

Baseline mean = 50, σ = 10, Δ = 2 (absolute), α = 0.05, power = 0.80.

n per group = 2 * ( (1.96 + 0.84)^2 * 10^2 ) / 2^2 = 392 per group.

Practical spreadsheet formulas (copy to Excel / Google Sheets)

Google Sheets / Excel snippets — assume inputs in cells:

• Alpha in B1 (e.g., 0.05), Power in B2 (e.g., 0.8)
• For proportions: p1 in B3, p2 in B4 (p2 = p1 + MDE)
• For means: sigma in B5, delta in B6

z quantiles (Sheets):

=NORM.S.INV(1 - B1/2)  // z_{1-alpha/2}

=NORM.S.INV(B2)       // z_{power}

Proportions sample size (per group) approximate (Sheets):

=LET(p1,B3,p2,B4,pbar,(p1+p2)/2,zA,NORM.S.INV(1-B1/2),zP,NORM.S.INV(B2),num,(zA*SQRT(2*pbar*(1-pbar))+zP*SQRT(p1*(1-p1)+p2*(1-p2)))^2, num/(p2-p1)^2)

Means sample size (per group):

=2*((NORM.S.INV(1-B1/2)+NORM.S.INV(B2))^2 * B5^2) / B6^2

Note: Excel uses NORM.S.INV similarly. If your spreadsheet doesn't support LET, break into helper cells.

Pre-registration template (paste into your experiment ticket)

1. Experiment objective
2. Primary metric and calculation
3. Population and eligibility
4. Assignment method (randomization method / strata / clusters)
5. MDE (absolute or relative)
6. Alpha and desired power
7. Planned sample size per arm (or fixed horizon)
8. Primary analysis method (test, regression adjustments, covariates)
9. Handling of missing data and outliers
10. Planned subgroup analyses (limited and pre-specified only)
11. Stopping rules and interim analysis plan
12. Rollout plan and guardrail metrics

Stopping rules (practical guidance)

• Avoid informal peeking. If you plan interim looks, pre-specify the number and use formal alpha-spending methods (e.g., O'Brien–Fleming) or adjust thresholds.
• For low-risk A/B tests you can run to pre-specified sample size or fixed time horizon.
• For high-risk interventions, use a Data Monitoring Committee and clearly defined safety triggers that cause immediate pause and investigation.

Risk & bias checklist

• Randomization integrity: verify assignment algorithm and logging.
• Cross-contamination: users seeing both variants?
• Instrumentation changes: did tracking or metric definitions change mid-test?
• Implementation fidelity: is treatment delivered consistently?
• Missingness or attrition: differential loss to follow-up by arm?
• Multiple comparisons: limit and pre-specify subgroup tests; correct for many tests.
• External validity: how will population/time/context affect rollout?
• Safety and ethics: any risk of harm, regulatory or privacy constraints?

Rollout plan & monitoring

Plan staged rollouts with clear criteria to proceed or roll back. Monitor these in near-real time during rollout:

• Primary metric by segment
• Key guardrail metrics (error rate, latency, complaints)
• Operational load (CPU, queue lengths, throughput)
• Implementation logs and anomaly alerts

Define rollout stages (e.g., 1% → 10% → 50% → 100%) and decision gates (time + no major guardrail breaches + consistent primary effect).

When to run experiments vs. observational analysis

Run randomized experiments when you can assign treatment and observe outcomes without unacceptable risk and when causal claims are needed. Use quasi-experimental or causal inference methods when randomization is infeasible or unethical, but expect stronger assumptions and validation steps.

Deliverables & outputs

• Pre-registered experiment plan (copy into the experiment ticket)
• Spreadsheet with sample-size calculations and inputs saved for traceability
• Post-experiment report: results, confidence intervals, subgroup checks, implementation notes, and decision (adopt/iterate/abandon)

Next steps & recommended tools

If you want interactive sample-size calculations inside the platform, consider creating an interactive form that collects the inputs above and stores them with each experiment (so teams can reproduce and compare). For live client-side calculation or server-side validated calculators you will need a small computation component beyond form rendering — see Capability notes below.

Quick reminder: pre-specification, realistic MDEs, and careful assignment/planning matter more than small improvements in statistical theory. Design experiments around decisions you will actually make.