Adaptive & Sequential Testing

Run experiments that learn while they run — allocate more exposure to promising options, stop sooner when evidence is clear, and keep decisions grounded in prespecified rules and sensible analysis.

Why this matters

Traditional fixed-sample A/B tests wait for a predetermined number of observations before choosing a winner. That can waste time and exposure on poor options. Adaptive and sequential testing techniques — including simple group-sequential rules, Bayesian stopping, and multi-armed bandit allocation — let you shift traffic, budget, or treatment intensity toward better performers as evidence accumulates. The result: faster learning, less harm from bad variants, and more efficient use of scarce experiment exposure.

What you will understand and practice

This resource helps you learn when and how to use adaptive approaches and what trade-offs they entail. You will be able to:

• Distinguish between fixed-sample, sequential, and bandit-style designs and pick the right family for your decision goals.
• Specify stopping rules, decision thresholds, and priors (when using Bayesian methods) before launching a test.
• Implement simple group-sequential tests and common bandit strategies (e.g., epsilon-greedy, Thompson Sampling) at a practical level.
• Measure and manage risks like inflated false positives, bias from unplanned peeking, and short-term overfitting to noisy metrics.
• Translate adaptive results into operational decisions, rollout plans, and follow-up validation tests.

Who benefits

Teams and organizations that need faster, lower-risk decisions across many contexts: product and growth teams running online experiments; service businesses testing messaging or pricing; manufacturers piloting process changes with limited production runs; nonprofits iterating fundraising appeals; healthcare operations testing scheduling workflows (without implying clinical use); and researchers or educators optimizing learning experiences. Adaptive testing is especially valuable when exposure is costly, time is limited, or many variants compete for attention.

Practical examples

Examples you can relate to:

• An e-commerce product team uses Thompson Sampling to allocate web sessions among three checkout flows, shifting traffic toward higher conversion while protecting statistical rigor with prespecified checks.
• A local services company tests three appointment reminder messages, stopping early for a clearly superior message to avoid continued cost from inferior reminders.
• A factory runs sequential experiments on a process parameter across shifts, using interim analyses to stop a harmful setting quickly and to focus time on promising adjustments.

When not to use adaptive allocation

Adaptive tests can bias some downstream analyses and may be inappropriate when you need complete, unbiased estimates of each variant’s long-run effect, when regulatory constraints demand fixed designs, or when small overall sample sizes make early allocation decisions unstable. Always prespecify decision rules, monitor experiment assumptions, and consider a final fixed holdout or replication to confirm results.

How this fits in the Discovery & Innovation Hub

This resource complements the Experimentation Playbook by focusing on designs that learn while running. Use the Primer and Quick Guide included here to understand core concepts and practical patterns, then consult the Playbook for experiment governance, metric selection, and analysis standards. Where helpful, THE platform capabilities — like Interactive Form rendering to capture experiment setup and results, JSON storage for experiment logs, or packaging an adaptive-testing collection as a reusable toolkit — can make your experiments easier to run, reproduce, and share across teams.