Experiment Plan Template

Experiment Plan Template

Use this template to design an experiment that produces reliable, defensible evidence. Fill each section before you run the experiment. Keep the plan short, specific, and versioned (add a plan ID and date). When possible preregister this plan so analysis choices are locked before seeing the data.

Basic info

• Plan ID & version: __________
• Date created: __________
• Owner(s): Name, role, contact
• Stakeholders: Decision maker(s) and sponsor(s)
• Expected duration: Start date — End date (or stopping rule)
• Context / background: Short rationale: why this experiment matters and what decision it will inform (2–4 sentences).

Problem / Opportunity

Describe the current situation, pain, or opportunity the experiment targets. Include relevant baseline numbers (conversion rate, defect rate, time to complete, throughput, NPS, etc.).

Baseline metric(s): __________ (value, measurement period)

Hypothesis

State a clear, testable hypothesis that connects an action to an expected measurable outcome. Use a format such as: “If we do X for population Y under conditions Z, then metric M will change by at least D in direction T.”

Example: "If we reduce the onboarding form to 3 fields for new users (X), then week-1 activation rate (M) will increase by at least 5 percentage points (D) for first-time sign-ups (Y)."

Primary and Secondary Metrics

List 1 primary metric (the main test outcome) and up to 3 secondary metrics (safety checks, leading indicators, or other outcomes to monitor).

• Primary metric: Name, definition, numerator, denominator, aggregation window (e.g., 7-day conversion rate)
• Secondary metric(s):
  1. Name + definition + why it's important (e.g., retention, error rate, revenue per user)
  2. ...

Minimum Detectable Effect (MDE) & Sample Size Guidance

Decide the smallest effect size that would change your decision (the MDE). If you cannot compute exact sample sizes now, use these pragmatic rules-of-thumb while arranging a formal power calculation:

• For proportion outcomes (A/B tests): a simple rule: smaller baseline rates require larger samples. Use a power calculator before running if possible.
• For continuous outcomes (time, score): 30+ independent observations per group is a bare minimum for central-limit approximations; 100+ is better for stable estimates when variability is high.
• If traffic or events are limited, consider longer test duration or sequential methods with prespecified interim checks.

Record your MDE, assumed baseline, desired power (commonly 80% or 90%), alpha (commonly 0.05), and calculated sample size per arm:

• Baseline: __________
• MDE: __________
• Power: __________
• Alpha: __________
• Sample size per group: __________

Experimental Design

Describe assignment method, arms, randomization unit, and blocking/stratification if any.

• Design type: A/B, multi-armed, factorial, stepped-wedge, cluster-randomized, before/after, etc.
• Arms: Control (describe), Treatment 1 (describe), Treatment 2 (if any)...
• Randomization unit: user / session / device / store / shift / batch
• Stratification / blocking: Yes / No; variables used
• Allocation ratio: e.g., 50:50, 60:40, 1:1:1

Instrumentation & Data Checklist

Ensure metrics are reliably measured before starting. Use this checklist and mark items complete.

• Define event names, schemas, and fields needed for metrics
• Tagging implemented in staging and production (Y/N)
• Data pipeline verified (ingestion, ETL, storage)
• Dashboards or reports built for primary & secondary metrics
• Logging for assignment, timestamps, and identifiers
• Fail-safe / fallback behavior defined if instrumentation fails
• Data owner and analyst assigned

Data Collection Plan

Describe what data will be collected, how often, where it will be stored, and who has access.

• Data variables: list key fields (user_id, group, timestamp, metric fields, covariates)
• Collection frequency: real-time / daily / weekly
• Storage location / table names: __________
• Retention policy: __________
• Access & privacy constraints: PHI/PII considerations, masking, approvals

Analysis Plan

Pre-specify the analysis methods to avoid fishing for significant results. Include the primary statistical test, covariate adjustments, and how missing data will be handled.

• Primary analysis method: e.g., difference-in-means t-test, logistic regression, Poisson regression, survival analysis
• Adjustment variables (covariates): list if any
• Transformations: log-transform, winsorize, etc.
• Handling outliers / missing data: describe rules (exclude, impute, sensitivity analyses)
• Pre-specified subgroup analyses: list subgroups and hypotheses (be conservative; treat as exploratory unless powered)
• Confidence intervals & effect sizes: report both; focus decisions on practical significance (effect size) as well as statistical significance

Stopping Rules & Interim Analysis

Define clear rules so the team knows when to stop or continue the experiment.

• Planned interim analyses? Yes / No. If yes, specify timing and adjustment method (e.g., alpha spending, Bonferroni, Bayesian update).
• Early stopping for harm: define thresholds (e.g., metric drop > X% with p < Y)
• Early stopping for overwhelming benefit: define threshold consistent with pre-specified error control
• Maximum duration or maximum sample size cap

Decision Criteria

Translate results into concrete decisions. Do not leave this ambiguous.

• If primary metric shows an effect ≥ MDE and CI excludes no effect: Recommend rollout / adopt.
• If effect is smaller than MDE or CI includes no effect: Recommend no-change / iterate on treatment.
• If harms on secondary metrics exceed predefined thresholds: Do not roll out; investigate causes.
• Provide a short decision matrix (outcome vs. recommended action).

Operational Runbook

Who will implement the experiment and what operational steps must be executed. Include rollback steps.

• Pre-launch checklist owner and sign-off
• Launch steps (feature flags, configuration changes)
• Monitoring during run (alerts, dashboards, contact list)
• Rollback procedures and responsible persons

Evidence & Learning Capture

Record outcomes, interpretation, and lessons so the experiment contributes to institutional learning.

• Final results summary (primary + secondary)
• Interpretation: what changed and why we think it happened
• Action taken (rollout, iterate, stop) and next steps
• Artifacts to save: datasets, analysis code, dashboards, meeting notes
• Date of decision and responsible approver

Quick Example (filled)

Baseline: 12% week-1 activation; Hypothesis: Simplified onboarding increases activation to 17% (MDE 5pp). Primary metric: week-1 activation. Design: A/B, 1:1 randomization by user. Sample size: 5,000 users per arm (calculated with power 80%, alpha 0.05). Instrumentation: event 'activation' verified in staging. Analysis: difference-in-proportions with 95% CI; missing users counted as not activated. Decision: If increase ≥5pp and CI does not include 0, rollout. If increase <5pp or harms to retention observed, iterate.

Notes & good practice

• Preregister the plan and avoid peeking at the primary outcome before analysis rules are fixed.
• Prefer practical effect sizes tied to decision value, not only statistical significance.
• Track and document surprises and implementation risk—some statistically positive results may be impractical to scale.
• When in doubt, run a pilot to validate instrumentation and assumptions before a full experiment.