CVR Rose 8%, but Profit per Visitor Fell 6%

SIMULATED / SAMPLE ANALYSIS — no client data

Hypothesis A promotional checkout variant increases conversion without reducing contribution profit per visitor.

Sample & method The simulated test contains 120,000 sessions, split evenly between control and variant (60,000 per arm). Conversion was tested with a two-proportion comparison. Contribution profit per visitor was evaluated with 10,000 stratified bootstrap resamples because order-level profit is skewed.

Evidence • Conversion rate: 4.00% → 4.32%; relative lift +8.0%. • Absolute CVR lift: +0.32 pp, 95% CI +0.09 to +0.55 pp; p = .006. • Contribution profit per visitor: $2.50 → $2.35; relative change −6.0%. • Bootstrap 95% CI for profit change: −9.4% to −2.1%. • Average order value: $72 → $66, consistent with a lower-margin order mix.

Uncertainty The CVR lift is unlikely to be sampling noise, but significance answers whether the metric changed—not whether the change was good for the business. Profit estimates are heavy-tailed, so the bootstrap interval and sensitivity to refunds and fulfillment costs matter more than a normal approximation.

Business decision REJECT THE VARIANT. Investigate whether the offer attracts lower-margin baskets, then redesign the incentive around contribution profit per visitor. A statistically significant conversion lift is not a win when unit economics deteriorate.