Dashboard significance is not the same as a defensible result. If you tested multiple variants, checked the numbers daily, or segmented after the fact, your reported p-value is probably wrong — usually in the direction that makes you ship something that doesn’t work.

Significance testing with the correct test for your metric type

Multiple-comparison correction — Holm-Bonferroni or Benjamini-Hochberg, chosen and justified. Four variants at 0.05 means roughly an 18% false-positive rate, not 5%

Bootstrap confidence intervals on the effect, so you see the plausible range rather than a single point estimate

Effect size — because statistically significant and commercially meaningful are different questions

Sequential-testing check — if the test was stopped early, I quantify what that did to your error rate

Guardrail and segment review — what moved that nobody looked at

A 2-3 page written readout with a plain-language recommendation, plus a technical appendix documenting every method and assumption

Arm-level counts (users and conversions per variant), or raw event data. CSV is fine. If you have assignment logs, include them — I’ll run a sample-ratio-mismatch check.

Turnaround: 5 business days.

Sample readouts available — ask and I’ll send two, including one where the “winner” fell apart under correction and one where the real finding was hiding in a segment.