How to interpret Statistical Significance
Statistical significance tells you the probability that the difference in conversion rates between your variations is real and not just due to random chance.
A confidence level of 95% means there is only a 5% chance that the result is a false positive. Most marketers wait for 95% before declaring a winner to avoid "Regret" (implementing a losing change).
The Problem with "Fixed Horizon" Testing
Traditional A/B testing requires you to wait until the end of the test to check results (no peeking!). If you check too early, you invalidate the math.
Zyro uses Sequential Bayesian testing, which allows you to check results daily without breaking the statistics. Better yet, our Bandit algorithms automatically route traffic to the winner while the test is still running.