Experiment Design Template & Power Calculator

What this tool is for

Use this template to design, run, analyze, and operationalize experiments that produce defensible causal evidence. It helps you frame the problem, select and protect a primary metric, choose assignment and sample-size parameters, pre-specify analysis and stopping rules, and prepare a safe rollout/rollback plan.

How to use it

1. Complete the Experiment Brief to align stakeholders on objective and primary metric.
2. Walk through the Assignment Method Checklist so randomization and unit of analysis are clean.
3. Use the Power Calculator examples to estimate required sample sizes. Replace example values with your baseline and minimum detectable effect (MDE).
4. Pre-specify stopping rules and the full analysis plan.
5. Document rollout and rollback procedures and guardrails.
6. After the experiment, complete the Post-Experiment Interpretation Checklist before acting on results.

1) Experiment brief (template)

Fill this at design time and save as the canonical experiment record.

• Title: [concise]
• Owner / Sponsor: [names & roles]
• Objective / Decision to be made: [What decision will this experiment inform?]
• Primary metric (unit + direction): [e.g., weekly active conversions, lower is better / higher is better]
• Rationale for primary metric: [tie to business goal]
• Secondary metrics / guardrails: [list with units and tolerance thresholds]
• Estimated baseline (control) value: [with measurement window & sample used to estimate it]
• Minimum Detectable Effect (MDE): [absolute or relative, and why it's meaningful operationally]
• Alpha / power: [recommended alpha=0.05 two-sided, power=0.8 default; justify changes]
• Planned sample size per arm: [from calculator]
• Allocation ratio: [1:1 typical, document if different]
• Start & end criteria: [date ranges OR sample-based stop]
• Stakeholders to notify: [ops, legal, product, analytics, support]

2) Assignment method checklist

• Unit of randomization defined (user, session, account, cluster). Ensure unit matches decision context.
• Randomization mechanism documented (hashing ID, random seed, service). Save seed/version.
• Blocking / stratification planned? (e.g., stratify by region or traffic source)
• Cluster randomization needed? If so, account for intra-cluster correlation (ICC) in sample size.
• Check for contamination / spillover risks and mitigation (e.g., cookies, multi-device users).
• Implementation fidelity test plan (how you will verify treatment delivered as intended).
• Data availability & latency: ensure metrics can be computed within the analysis window.
• Pre-register experiment brief and analysis plan where your organization keeps records.

3) Power calculator (worked examples & formulas)

Two common scenarios are shown: binary outcomes (proportions, e.g., conversion rate) and continuous outcomes (means). Use the worked examples to understand scale before creating an exact calculation with your analytics tools.

Binary outcome (two-arm, equal allocation)

Notation: p1 = baseline conversion (control), p2 = expected conversion (treatment), MDE = p2 - p1, p̄ = (p1 + p2)/2.

Approximate sample size per arm (normal approximation):

n ≈ [ (Z_{1-α/2} * sqrt(2*p̄*(1-p̄)) + Z_{power} * sqrt(p1*(1-p1) + p2*(1-p2)) )^2 ] / (p2 - p1)^2

Recommended defaults: alpha 0.05 (Z_{1-α/2}=1.96), power 0.8 (Z_{power}=0.84).

Example 1

Baseline p1 = 10% (0.10). MDE = +2 percentage points → p2 = 0.12. Using usual Z values, the calculator gives approximately n ≈ 3,835 per arm (≈ 7,670 total).

Example 2

Baseline p1 = 20% (0.20). MDE = +2 percentage points → p2 = 0.22. Approximate n ≈ 6,500 per arm (≈ 13,000 total).

Notes:

• Smaller absolute MDEs require much larger samples.
• For rare events (very low baseline), consider Poisson or exact methods and/or longer measurement windows.
• When allocation is unequal, multiply n by (1 + 1/k)/2 where k is the allocation ratio (see stats reference).

Continuous outcome (difference in means)

Notation: σ = standard deviation (assumed equal), δ = detectable difference (mean_treatment - mean_control).

Approximate sample size per arm:

n ≈ 2 * (Z_{1-α/2} + Z_{power})^2 * σ^2 / δ^2

Example

If σ ≈ 10 units and you want to detect δ = 3 units, then with alpha 0.05 and power 0.8:

Z-sum ≈ 1.96 + 0.84 = 2.80 → (2.80)^2 ≈ 7.84 → n ≈ 2 * 7.84 * 100 / 9 ≈ 175 per arm.

Clustered designs

If randomization is by cluster (e.g., by store), inflate sample size by the design effect: DE = 1 + (m - 1) * ICC, where m = average cluster size and ICC is intra-cluster correlation. Multiply the per-arm sample by DE and then compute number of clusters required.

4) Stopping rules & pre-analysis plan (template)

Pre-specify all of the following and save a timestamped copy:

• Primary hypothesis: precise null and alternative statements for the primary metric.
• Primary analysis method: test type (t-test, proportion z-test, regression), covariates to include, clustering adjustments, and unit of analysis.
• Data cleaning rules: event deduplication, outlier handling, participant exclusions and reasons.
• Missing data: expected patterns and handling approach (e.g., intent-to-treat, imputation rules).
• Multiple comparisons: correction approach or explicit list of confirmatory comparisons versus exploratory analyses.
• Stopping rules:
  • Fixed-sample (preferred): stop when pre-specified sample or time reached.
  • Sequential testing: if used, specify alpha spending function (e.g., O'Brien-Fleming) or correction method to control Type I error.
  • Safety/harm rules: immediate stop if a guardrail metric crosses a pre-specified threshold for harm.
• Exploratory / subgroup analysis plan: list specific subgroups pre-specified for confirmatory analysis; everything else is exploratory.
• Primary reporting format: effect estimate with 95% CI, p-value, sample sizes, pre-specified graphs (e.g., cumulative metric over time), and context notes.

5) Rollout & rollback plan (template)

Define how you'd move from experiment to production and how to revert safely if needed.

• Rollout criteria: minimal effect size, statistical significance, operational readiness, stakeholder sign-offs.
• Phased rollout plan: percentage-based ramp, monitoring windows, who approves each phase.
• Monitoring during rollout: list of metrics and dashboards refreshed at intervals, alert thresholds, on-call owner.
• Rollback triggers: severe degradation of primary metric, breach of guardrails, operational incidents, or unexpected safety concerns.
• Rollback procedure: step-by-step technical rollback, communications to customers and teams, verification steps after rollback.
• Post-rollout audit: confirm feature behaves as expected in production and re-run key sanity checks.

6) Post-experiment interpretation checklist

Complete this before making irreversible operational changes.

1. Confirm the pre-registered analysis was followed exactly; document any deviations and rationale.
2. Check randomization balance on key covariates; if imbalance exists, consider adjusted analyses.
3. Verify implementation fidelity: were treatments delivered to the intended units?
4. Inspect data for missingness, outliers, duplicate events, and unexpected filtering effects.
5. Examine effect size and 95% CI — is the observed difference practically meaningful, not just statistically significant?
6. Review secondary metrics and guardrails for unexpected harms or compensatory effects.
7. Check for p-hacking or selective reporting: ensure all pre-specified outcomes are reported and exploratory analyses are labeled as such.
8. Assess external validity: will the effect likely hold when scaled or in other segments?
9. Operational readiness: can teams support the change? Are there deployment or customer-support implications?
10. Decision: adopt, iterate (A/B/n or follow-up experiments), roll out gradually, or reject. Document the decision and next steps.

Common pitfalls & guidance

• Underpowered tests: avoid interpreting non-significant results as proof of no effect when MDE was large relative to realistic change.
• Fishing expeditions: pre-specify your primary outcome and limit confirmatory subgroup tests.
• Ignoring implementation: a correct statistical result is useless if the feature wasn't delivered as intended.
• Over-interpretation of early peeks: sequential testing requires correction to maintain Type I error control.
• Confusing statistical with operational significance: small percent changes may be statistically real but operationally irrelevant.

Appendix — Quick reference formulas

Binary outcomes (normal approx): see section above for full formula. Continuous outcomes (difference in means): n ≈ 2 * (Z_{1-α/2} + Z_{power})^2 * σ^2 / δ^2. For cluster designs, multiply by design effect DE = 1 + (m - 1) * ICC.

Suggested next steps & capability notes

This HTML template is deliberately self-contained so teams can start designing experiments immediately. To make this tool more useful inside The Hunger Engine, consider the following platform enhancements (not yet implemented in this content item):

• Convert the experiment brief and analysis-plan templates into an Interactive form so teams can save experiment records (uses Interactive Form Rendering and Content Data Submission capabilities).
• Add an interactive power calculator that accepts baseline, MDE, alpha, power, allocation ratio, and ICC (when relevant) and returns sample sizes. Persist inputs and results to experiment records.
• Provide automated checks that validate pre-registered analysis against the executed analysis and flag deviations.
• Package as a reusable toolkit within the domain so teams can clone and tailor experiment templates to site-specific standards.

If you want, I can produce an InteractiveForm version of this template that collects experiment parameters, computes sample sizes for you, and saves completed experiment briefs into the platform (requires enabling the Interactive Form Rendering and Content Data Submission capabilities).