Your A/B test said "winner" — but did revenue per visitor move?
CVR is the metric everyone watches. It's easy to track and moves fast. But CVR going up doesn't mean the variant made you more money.
I've seen checkout tweaks that lifted CVR 8% while AOV dropped enough to net a 6%...
What makes an A/B test readout useful to a product team?
My preferred first page answers four questions:
What changed, and by how much?
How uncertain is the estimate?
Did an important guardrail get worse?
What decision does the evidence support, and what remains unresolved?
A result...
Five arms. One has p=.03, but Benjamini–Hochberg gives q=.15. What should the team do?
1 voted
100%
0 voted
0%
1 vote
Closed
CVR rose 8%, but contribution profit per visitor fell 6%. The result is statistically significant. What should the team do?
1 voted
50%
1 voted
50%
2 votes
Closed
Most A/B tests don’t fail during analysis. They fail before launch.
Common causes:
• The sample is too small
• The minimum worthwhile effect was never defined
• There is no stopping rule
• Too many metrics are treated as primary
• Important guardrail metrics are missing
The result...
contra.com
Pre-Launch A/B Test Review — Power & Sample Size Check by Mo Rahman
Read more about Pre-Launch A/B Test Review — Power & Sample Size Check by Mo Rahman on Contra.
