How - Gather
Power Analysis - Checkout
Hypothesis/Success Criteria: If we clearly call out the guest checkout option then we will increase conversion by at least 2%.
What is the optimal sample size for the given hypothesis?
Sample Size (n) = Unknown?
Effect Size (d) = 2%
Power = 80%
Sig Level (alpha/confidence level) = 0.05 or 95%
Sample Size = 19,625
This tells us that we need ~20k sessions to reach 95% confidence to see a 2% increase in conversion at an 80% probability that the detected lift actually exists. If we do not reach a 2% increase in conversion at 95% conf. in the optimal sample size then we failed to reject the null hypothesis.
If we met the 2% increase in conversion rate at 95% confidence in ~20k then we would have rejected the null hypothesis.