Case study: two small experiments, two different decisions

Here are short, realistic scenarios showing how design choices change with context.

Bench lab — limited reagents, high cost per run

Hunger: Decide whether changing buffer composition produces a meaningful activity improvement worth further development.

Design highlights:

• Primary outcome: initial reaction rate measured in µmol/min at 30s (exact protocol recorded).
• MDE set to the smallest improvement that would justify additional optimization work (e.g., ≥15% increase).
• Pilot: run 6 samples per condition to estimate variability, then re-evaluate sample needs.
• Controls: buffer-only negative and a known activator positive control on every plate.
• Pre-registration: save the analysis plan and success criteria to project records to avoid selective reporting.

Product A/B test — large user base, fast signal

Hunger: Determine if a small UI change increases click-through enough to justify rollout.

Design highlights:

• Primary outcome: click-through rate within 24 hours (binary metric).
• MDE tied to business impact (e.g., a 1% absolute increase in CTR that improves revenue enough to cover cost of change).
• Large user pool allows a pre-specified sample-size calculation and short sequential checks with formal stopping rules to avoid inflated false positives.
• Randomization and automated logging reduce measurement bias; analysis plan details demographic adjustment to avoid confounding.

Shared lessons

• Make the decision you care about the anchor for the design.
• Record the analysis plan before looking at the primary data.
• Use a pilot when variance is uncertain; treat exploratory findings as hypothesis-generating unless the study was powered and pre-registered.