The network for creativity
Join 1.25M professional creatives like you
Connect with clients, get discovered, and run your business 100% commission-free
Creatives on Contra have earned over $150M and we are just getting started
Back to feedPost
Most A/B tests don’t fail during analysis. They fail before launch.
Common causes:
• The sample is too small
• The minimum worthwhile effect was never defined
• There is no stopping rule
• Too many metrics are treated as primary
• Important guardrail metrics are missing
The result is weeks of traffic and a dashboard nobody can confidently act on.
Before launch, I calculate the required sample size, expected runtime, minimum detectable effect, statistical power, and guardrails—then provide a clear go/don’t-go recommendation.
If your SaaS or e-commerce team has an experiment planned, I offer a three-day pre-launch review:
contra.com
Pre-Launch A/B Test Review — Power & Sample Size Check by Mo Rahman
Read more about Pre-Launch A/B Test Review — Power & Sample Size Check by Mo Rahman on Contra.
The network for creativity
Join 1.25M professional creatives like you
Connect with clients, get discovered, and run your business 100% commission-free
Creatives on Contra have earned over $150M and we are just getting started
Challenges
View all
Related posts
What makes an A/B test readout useful to a product team?
My preferred first page answers four questions:
What changed, and by how much?
How uncertain is the estimate?
Did an important guardrail get worse?
What decision does the evidence support, and what remains unresolved?
A result can be statistically significant and still too small to matter. An inconclusive result can still leave a meaningful gain or loss plausible. The decision needs more than a green badge.
I built an independent A/B analyzer to make these checks easier to inspect. The project includes a live demo, downloadable reports and documented limitations:
https://contra.com/p/mgoXb3qY-ab-test-analyzer-correct-p-values-power-and-fdr
If you work on product or CRO, what do you wish analysts put on the first page more often?
This is a strong framing of experiment readouts: separating signal, uncertainty, guardrails, and the actual decision keeps the team honest. The reminder that significance is not the same as usefulness is especially important.
