Five Arms, One Nominal Winner, Zero Valid Launches

SIMULATED / SAMPLE ANALYSIS — no client data

Hypothesis At least one of four onboarding variants improves activation conversion versus control.

Sample & method The simulated test contains 50,000 sessions, split evenly across one control and four variants (10,000 per arm). Each variant was compared with control, and family-wise error was controlled across four comparisons with Holm correction.

Evidence • Control conversion: 10.00%. • Best variant conversion: 10.87%; absolute lift +0.87 pp. • Unadjusted result: p = .0438. • Holm-adjusted result: p = .175. • Multiplicity-adjusted confidence interval for the best arm: −0.21 to +1.95 pp. • Exploratory mobile interaction: +1.6 pp, 95% CI +0.3 to +2.9 pp.

Uncertainty The global test does not support a launch after correcting for four chances to find a winner. The mobile result was discovered after segmentation, so it is hypothesis-generating rather than confirmatory.

Business decision DON’T SHIP GLOBALLY. Turn the mobile interaction into a preregistered follow-up experiment with one primary comparison and adequate power. The useful outcome is a sharper next hypothesis, not a weak launch claim.