A/B Testing Power & Sample Size Calculator (Interactive Sample Plan)

{ "Title": "A/B Testing Power & Sample Size Calculator", "IntroductionHtml": "

Design tests that can actually answer your question

This tool helps you build a clear, reproducible sample-size and timing plan for common A/B experiments. Enter your baseline (conversion rate or mean and SD), choose a minimum detectable effect (MDE), and set statistical choices such as power, significance, sidedness, allocation and traffic. The form saves a complete test plan you can share, export, or run through an automatic calculator when your team enables the calculation engine.

Why this matters: Underpowered tests or vague MDEs lead to wasted time and misleading null results. Being explicit about assumptions (baseline, MDE, allocation) helps you interpret outcomes and avoid quietly post‑hoc rationalization.

Quick guidance and formulas

For two-sample proportions (equal allocation) a commonly used approximate formula for the sample size per group is:

n ≈ [ (Z_{1-α/2} * sqrt(2 * p̄ * (1 - p̄)) + Z_{power} * sqrt(p1*(1-p1) + p2*(1-p2)))^2 ] / (p1 - p2)^2

Where p1 is baseline, p2 = p1 ± MDE, and p̄ = (p1 + p2)/2. For continuous outcomes (means) you need an estimate of the standard deviation and can use the familiar two-sample t-test approximations.

The tool supports both conversion-rate (proportion) and continuous-mean scenarios, along with options for absolute/relative MDE and one- or two-sided tests.

Practical cautions

• Peeking & sequential testing: Repeatedly checking results inflates false positives. Pre-specify stopping rules or use sequential methods and appropriate corrections.
• Multiple comparisons: If you test many variants, adjust significance (e.g., Bonferroni, BH) or use multi-arm methods to avoid excess Type I error.
• External validity: Ensure the experiment environment reflects the population you want to generalize to; implementation differences can invalidate results.

Using this form

Fill required fields and press Calculate & Save Plan. The platform will store the plan for your team. If the calculation engine is available, the form will also compute sample sizes and estimate duration from traffic inputs; otherwise use the guidance and formula above or export the inputs as CSV for an offline calculation.

", "SubmitLabel": "Calculate & Save Plan", "SuccessMessage": "Plan saved. If the platform calculation engine is enabled, computed sample sizes and duration will appear here. Otherwise export the plan as CSV for offline calculation.", "DataType": "ab-test-sample-plan", "SchemaVersion": "1.0", "Fields": [ { "Key": "test_name", "Label": "Test name", "FieldType": "text", "Required": false, "HelpText": "A short descriptive name for this experiment (e.g., 'Homepage CTA color test')." }, { "Key": "test_type", "Label": "Outcome type", "FieldType": "select", "Required": true, "HelpText": "Choose whether the primary metric is a conversion (proportion) or a continuous metric (mean).", "Options": [ {"Value": "proportion", "Label": "Two-sample proportion (conversion)"}, {"Value": "continuous", "Label": "Two-sample mean (continuous)"}, {"Value": "multiarm", "Label": "Multi-arm (ANOVA / multiple variants) - planning only"} ], "DefaultValue": "proportion" }, { "Key": "baseline_rate", "Label": "Baseline conversion (p1) — percent", "FieldType": "number", "Required": false, "HelpText": "Enter baseline conversion as a percentage (e.g., 4.5 for 4.5%). Required for proportion tests.", "DefaultValue": 5 }, { "Key": "baseline_mean", "Label": "Baseline mean (for continuous outcome)", "FieldType": "number", "Required": false, "HelpText": "Enter the baseline mean value for continuous outcomes (e.g., average order value).", "DefaultValue": null }, { "Key": "baseline_sd", "Label": "Baseline standard deviation (for continuous outcome)", "FieldType": "number", "Required": false, "HelpText": "Estimate of the standard deviation for the continuous metric. Required for continuous tests.", "DefaultValue": null }, { "Key": "mde_type", "Label": "MDE type", "FieldType": "radio", "Required": true, "HelpText": "Choose whether your Minimum Detectable Effect is an absolute difference or a relative percent change.", "Options": [ {"Value": "absolute", "Label": "Absolute (percentage points for conversions, absolute units for means)"}, {"Value": "relative", "Label": "Relative (% change from baseline)"} ], "DefaultValue": "relative" }, { "Key": "mde_value", "Label": "Minimum Detectable Effect (MDE)", "FieldType": "number", "Required": true, "HelpText": "If relative, enter the percent change you care about (e.g., 10 for 10%). If absolute and using conversions, enter percentage points (e.g., 1.2 for +1.2%).", "DefaultValue": 10 }, { "Key": "alpha", "Label": "Significance level (alpha)", "FieldType": "number", "Required": true, "HelpText": "Common values: 0.05. Use smaller alpha for stricter control of false positives or when doing many tests.", "DefaultValue": 0.05 }, { "Key": "power", "Label": "Desired power", "FieldType": "number", "Required": true, "HelpText": "Probability of detecting the MDE if it truly exists. Common values: 0.8 or 0.9.", "DefaultValue": 0.8 }, { "Key": "sided", "Label": "Sidedness", "FieldType": "select", "Required": true, "HelpText": "Choose one- or two-sided test. Two-sided is default unless you have a strong directional hypothesis.", "Options": [ {"Value": "two", "Label": "Two-sided"}, {"Value": "one", "Label": "One-sided"} ], "DefaultValue": "two" }, { "Key": "variants", "Label": "Number of variants (including control)", "FieldType": "number", "Required": true, "HelpText": "For a simple A/B test enter 2. For multi-arm tests enter the total number of arms. Multi-arm calculations are approximate here.", "DefaultValue": 2 }, { "Key": "allocation_ratio", "Label": "Allocation ratio (control : variant)", "FieldType": "number", "Required": false, "HelpText": "Relative allocation per variant. For equal allocation in a 2-arm test use 1 (control:variant). For 50/50 split between two arms use 1. To allocate more traffic to control use >1.", "DefaultValue": 1 }, { "Key": "daily_traffic", "Label": "Average visitors per day (total to testable population)", "FieldType": "number", "Required": false, "HelpText": "Helps estimate calendar duration. Enter the total number of test-eligible visitors per day (not per arm).", "DefaultValue": 10000 }, { "Key": "expected_uplift_direction", "Label": "Expected uplift direction", "FieldType": "select", "Required": false, "HelpText": "Whether you expect an increase or decrease. Useful for interpreting relative MDE inputs.", "Options": [ {"Value": "increase", "Label": "Increase"}, {"Value": "decrease", "Label": "Decrease"}, {"Value": "either", "Label": "Either (two-sided)"} ], "DefaultValue": "increase" }, { "Key": "notes", "Label": "Experiment notes / constraints", "FieldType": "textarea", "Required": false, "HelpText": "Any implementation constraints, segments, or seasonal concerns to record in the plan.", "DefaultValue": "" }, { "Key": "export_csv", "Label": "Export plan as CSV after saving", "FieldType": "yesno", "Required": false, "HelpText": "Choose yes to save the plan and prepare a CSV of the input fields for offline calculation or sharing.", "DefaultValue": "no" } ] }