Back to projects
building
Checkout Conversion A/B Test
Did the new checkout actually convert better?
An end-to-end experiment analysis: control vs. a redesigned checkout, with conversion rate as the primary metric. The depth is in doing it right — a power analysis that fixes the sample size before any data, sanity checks for sample-ratio mismatch and data-quality traps, a two-proportion z-test with confidence intervals, and an honest read on whether the lift is practically, not just statistically, significant.
how.it.works
How it works
01Designhypothesis, primary metric, guardrails, and a power analysis that sets the required sample size before any data is touched.
02Data qualityresolve the group/landing-page mismatch, dedupe users, and run a sample-ratio-mismatch (SRM) check.
03Analyzeconversion lift with a two-proportion z-test, a chi-square cross-check, and a 95% confidence interval.
04Decidestatistical vs practical significance, novelty over time, and a clear ship / no-ship recommendation.
The interactive demo for this project is in progress.