Experimentation Design & Analysis Template

Purpose

Use this template to pre-specify an experiment so that results are defensible, actionable, and easy to operationalize. Filling this out before launching reduces ambiguity, prevents common analysis mistakes, and speeds safe rollout decisions.

Context & Problem Statement

Describe the business or operational problem the experiment addresses, why it matters, the decision the team must make if the result is positive or negative, and any constraints (time, budget, safety, regulatory).

Example: Improve onboarding completion so new users reach the activation milestone within 7 days. If successful, roll out to all new users; if not, iterate on the intervention.

Hypothesis

Write a clear, testable hypothesis that links an intervention to an expected measurable change.

Template: "If we apply intervention X to population Y, then primary metric will change by at least the Minimum Detectable Effect (MDE) compared with control during the measurement window, because rationale."

Example: If we show a streamlined onboarding checklist to new users, then the 7-day activation rate will increase by at least 3 percentage points (from 20% to 23%) because clearer steps reduce friction.

Primary and Guardrail Metrics

• Primary metric (one): The main outcome used to make the decision (define precisely, unit, denominator, measurement window).
• Secondary metrics: Important supporting outcomes (ranked). These are for learning, not decision unless pre-specified.
• Guardrail metrics: Safety, quality, or business metrics that must not be harmed. Specify thresholds that would block rollout.

Example: Primary = 7-day activation rate (users who complete activation within 7 days). Guardrail = 7-day retention (%) should not drop by more than 1 percentage point.

Minimum Detectable Effect (MDE) & Sample Size

Choose an MDE that reflects a meaningful business impact and is realistic given resource constraints. Document the baseline rate, desired MDE, statistical power (commonly 80% or 90%), alpha (commonly 0.05), and one- or two-sided test.

Include a link to a power/sample-size calculator for transparency. Example calculators: Evan Miller AB sample size or your organization's standard tool.

Example inputs: Baseline 20%, MDE 3 percentage points, power 80%, alpha 0.05, two-sided => required N per group = 18,000 (document exact calculator and assumptions).

Note on trade-offs: Lowering MDE or increasing power increases sample size. If required sample size is impractical, consider higher-impact treatments, alternative metrics, or longer test duration.

Assignment & Randomization

Explain the unit of randomization (user, account, session, device), the randomization method, hashing/seed approach, stratification or blocking (if any), and how you will prevent leakage or contamination.

Specify any exclusion criteria or eligibility rules and how they are enforced.

Example: Randomize at the user ID level using the system-wide hashed-random bucket with seed 2026-08-01; stratify by country for top 3 countries; exclude users created before feature release.

Instrumentation & Logging Checklist

List the events, properties, and logs that must be recorded reliably for the experiment. Include ownership and verification steps.

• Event for treatment exposure (who saw/received the intervention).
• Primary and secondary outcome events with timestamps.
• Assignment bucket recorded with each relevant event.
• Client/server-side logging and fallback to ensure loss-resilience.
• Data pipeline owners and verification plan (smoke tests, test users, replay logs).

Pre-launch checklist: Run a short QA: known test users, verify treatment assignment, validate event counts, check sampling rates, and confirm assignment persists when appropriate.

Pre-specified Analysis Plan

Document exactly how the data will be analyzed. Keep the plan simple and concrete to avoid post-hoc choices.

• Analysis population: Intent-to-treat (ITT), per-protocol, or other. Define precisely.
• Primary comparison: Statistical test (t-test, logistic regression, difference-in-means), adjustment covariates (pre-specified), and treatment of clustering.
• Handling missing data: Define imputation or exclusion rules.
• Multiple comparisons: If multiple primary tests, specify correction (Bonferroni, Benjamini-Hochberg) or hierarchical testing order.
• Secondary analyses: List planned subgroup analyses and interactions (limit number to reduce noisy hunting).
• Robustness checks: Pre-specify a small set of sensitivity analyses (e.g., alternative windows, trimming outliers).

Reporting: List point estimates, confidence intervals, effect sizes, sample sizes per arm, p-values, and metric denominators. Prefer confidence intervals and business-significant interpretation over p-value-only claims.

Stopping Rules & Interim Analyses

If you plan interim looks, pre-specify timing, stopping boundaries, and the statistical method (alpha spending, group sequential methods). If you will not do interim looks, state that explicitly.

Example: No interim peeking. Final analysis after N per arm is reached. If unsafe outcomes appear in guardrails (e.g., retention drop >1 pp with p<0.01), pause and investigate immediately.

Rollout & Rollback Strategy

Describe how a positive or negative result will be operationalized, including phased rollouts, monitoring after rollout, and rollback criteria.

Include owners for rollout, monitoring dashboards, alert thresholds, and a timeline for decision and full deployment.

Example: If primary metric shows a statistically and practically significant improvement and guardrails are OK, roll out to 25% for 1 week and monitor guardrails before full rollout. If guardrails breach threshold at any point, roll back immediately and open a post-mortem.

Ethics, Safety & Compliance

Note any ethical, privacy, accessibility, or regulatory issues. Document informed-consent needs, data retention limits, PII handling rules, and approvals required.

Ownership, Timeline & Resources

• Experiment owner (decision maker)
• Data owner / analyst
• Engineering owner (instrumentation & rollout)
• Estimated start date, expected end date or required sample size target, and resources required

Worked Example (Filled Template)

Context: New-user onboarding completion is low; increasing activation increases first-week retention value.

Hypothesis: Showing a streamlined checklist during signup will increase 7-day activation from 20% to at least 23% (MDE = 3 pp), enabling improved downstream retention.

Primary metric: 7-day activation rate (binary per user).

Guardrail: 7-day retention should not fall by more than 1 pp.

MDE & Sample Size: Baseline 20%, MDE 3 pp, power 80%, alpha 0.05, two-sided. Using external calculator => ~18,000 users per arm. (Document calculator URL and exact parameters.)

Assignment: User-level randomization via hashed user ID; stratify by country for top 3 countries; exclude internal test accounts.

Instrumentation: Events: signup_complete, checklist_shown (treatment exposure), activation_complete (outcome). Assignment bucket included on each event. QA passed on test users.

Analysis plan: ITT comparison of proportions using logistic regression adjusted for country; report difference in proportions and 95% CI; no interim looks; planned subgroup: new users on Android (single prespecified subgroup).

Rollout: If positive and guardrails OK, roll to 25% for 7 days with monitoring; if stable, full rollout; rollback immediately if guardrail threshold violated.

Quick Checklist Before Launch

1. Hypothesis and decision criteria written and approved.
2. Primary and guardrail metrics precisely defined and instrumented.
3. Sample size calculation performed and logged.
4. Randomization implemented and verified for edge cases.
5. Logging verified for treatment exposure and outcomes.
6. Analysis plan pre-registered or stored with the experiment record.
7. Owners and rollout/rollback plan assigned and communicated.
8. Ethical and compliance approvals obtained if required.

Where to Save & How to Use

Save this completed template with the experiment record before launch. Attach instrumentation smoke-test results, sample-size calculation screenshot/URL, and a link to the analysis notebook or dashboard. Use stored records to speed future replications and meta-analysis.

Further Reading & Tools

• Simple power/sample-size calculator: Evan Miller AB sample size
• Good practice references: pre-registration, analysis plans, and sequential testing methods (link organizational resources where available).

Notes & Common Pitfalls

• Avoid multiple unplanned peeks at the data; pre-specify any interim analyses.
• Report effect sizes and confidence intervals, not just p-values.
• Guard against data-loss or unequal instrumentation across arms.
• Be cautious interpreting small, statistically significant effects with little practical impact.