TrafficLoopback
Sign in / Sign up

A/B Test Significance Calculator

Is your A/B test result real, or just noise?

Enter your A/B test results to see if the difference is statistically significant or just noise. We use a two-proportion z-test — the same math behind most A/B testing tools — and translate the output into plain English.

Variant A (Control)

234 / 4,500 = 5.20%

Variant B (Treatment)

285 / 4,500 = 6.33%

Use two-tailed unless you decided in advance that you only care about B beating A.

Statistically significant
Variant B beats A with 95% confidence.
Variant A rate
5.20%
234 / 4,500
Variant B rate
6.33%
285 / 4,500
Absolute lift
+1.13 pp
Relative lift
+21.79%
p-value
p = 0.0211
z-statistic
2.306
95% CI on absolute lift[+0.17 pp, +2.10 pp]
95% CI on relative lift[+3.28%, +40.31%]
Conversion rates with 95% confidence intervals
0.0%2.0%4.1%6.1%8.1%5.20%Variant A6.33%Variant B
Reasoning trace
p_A = 0.052000, p_B = 0.063333
p_pool = 0.057667
SE_pooled = 0.004914
SE_unpooled = 0.004913
z = (p_B − p_A) / SE_pooled = 2.306132
two-tailed p = 2·(1 − Φ(|z|)) = 2.1103e-2
alpha = 0.0500 → reject H0

Significance testing assumes a fixed sample size set in advance. If you've been peeking at results and stopped when you saw significance, your real false-positive rate is higher than the stated alpha. Sequential testing or a bandit avoids this trap.

Skip the math. Let a bandit run your tests for you.

TrafficLoopback runs Thompson-sampling bandits on your owned traffic, so you find winning ad creatives without manual significance tests. Free during beta.

Start free in beta
More free tools
Bandit Simulator
Watch a multi-armed bandit beat a 50/50 split — live.
Sample Size Calculator
How many visitors do you need before you can call a winner?
Break-Even Calculator
What CPC can you actually afford to pay?