A/B Testing and Statistical Significance
Run an A/B test by hand: a two-proportion z-test, a p-value, a confidence interval, the sample size you need up front, and a simulation of why peeking at results early inflates false positives to 20%.
An A/B test randomly splits users between a control and a change, then tests whether a metric differs by more than chance would explain, usually with a two-proportion z-test. Interviewers expect the sample size to be fixed in advance: checking ten times and stopping at p below 0.05 raised false positives from 4.8% to 19.7%.
An A/B test compares two versions by splitting users at random and testing whether a metric differs by more than chance would explain. A two-proportion z-test gives the p-value, a confidence interval shows the size of the effect, and a sample size calculation done before the test says how long to run it. Stopping early when the result looks good breaks the guarantees.
- In the worked A/B test, 200 of 2,000 users converted on A (10%) and 240 of 2,000 on B (12%), giving z = 2.021 and a two-sided p-value of 0.0432.
- The 95% confidence interval for the lift from 10% to 12% with 2,000 users per arm ran from 0.06 to 3.94 percentage points, so the effect could be tiny.
- Detecting a move from 10% to 12% conversion at alpha 0.05 and power 0.8 needs about 3,841 users in each group.
- In 2,000 simulated A/A tests, stopping at any of 10 looks when p fell below 0.05 gave 19.7% false positives, versus 4.8% when looking only once at the end.
- A p-value is the probability of a result at least as extreme if the null hypothesis were true; it is not the probability that the null is true.
- Required A/B test sample size grows roughly with the inverse square of the effect size, so halving the lift to detect roughly quadruples the users needed.
01The setup
In an A/B test, users are randomly assigned to version A (control) or version B (the change). Random assignment is what makes the comparison fair, because on average the two groups differ only in the change. You then measure a metric, here a conversion rate, and ask whether the difference could just be luck.
Our example: A has 200 conversions out of 2,000 users (10%). B has 240 out of 2,000 (12%). B looks better by two percentage points. Is that real?
02The hypothesis test
The null hypothesis says both versions have the same true conversion rate. The p-value is the probability of seeing a gap at least this large if that were true. Small p-value means the data is hard to explain by chance.
For two proportions we pool the rates under the null, compute a standard error, and form a z-score.
import numpy as np
from scipy import stats
conv_a, n_a = 200, 2000
conv_b, n_b = 240, 2000
pa, pb = conv_a / n_a, conv_b / n_b
p = (conv_a + conv_b) / (n_a + n_b)
se = np.sqrt(p * (1 - p) * (1 / n_a + 1 / n_b))
z = (pb - pa) / se
pv = 2 * (1 - stats.norm.cdf(abs(z)))
print(round(z, 3), round(pv, 4)) # 2.021 0.0432z is 2.021 and the two-sided p-value is 0.0432. With the usual cutoff of 0.05 this counts as significant, but only just. A p-value of 0.043 does not mean there is a 4.3% chance the result is wrong, and it does not mean B is 95.7% likely to be better. It says that if the versions were truly identical, a gap this large would show up about 4% of the time.
03Effect size: the confidence interval
se2 = np.sqrt(pa * (1 - pa) / n_a + pb * (1 - pb) / n_b)
low = pb - pa - 1.96 * se2
high = pb - pa + 1.96 * se2
print(round(low, 4), round(high, 4)) # 0.0006 0.0394The 95% interval for the lift runs from 0.06 to 3.94 percentage points. The lower end is barely above zero. So the honest summary is: B probably helps, but the true gain could be tiny or nearly four points. If a decision depends on the gain being at least one point, this test does not settle it. Report the interval alongside the p-value, because a significant result with a wide interval is weak evidence about size.
04Statistical versus practical significance
With millions of users, a 0.05 point lift can produce a very small p-value. That tells you the effect is real, not that it is worth the engineering cost or the added complexity. Decide in advance what size of change would justify shipping. This is the minimum detectable effect, and it feeds into the sample size calculation below.
05Plan the sample size first
Sample size depends on the baseline rate, the smallest lift you care about, the significance level (usually 0.05) and the power (usually 0.8, the chance of detecting a real lift of that size).
za, zb = 1.959964, 0.841621 # alpha 0.05 two-sided, power 0.8
p1, p2 = 0.10, 0.12
bar = (p1 + p2) / 2
n = ((za * np.sqrt(2 * bar * (1 - bar)) + zb * np.sqrt(p1 * (1 - p1) + p2 * (1 - p2))) ** 2) / (p1 - p2) ** 2
print(round(n)) # 3841To detect a move from 10% to 12% you need about 3,841 users in each group. Our test used 2,000 each, so it was underpowered for a two-point lift. That fits what we saw: borderline significance and a wide interval. Halving the lift you want to detect roughly quadruples the users required, which is why small effects are expensive to find.
06The peeking problem
Suppose both versions are identical (an A/A test) and you check the result ten times as data arrives, stopping if p drops below 0.05 at any check. Each look is another chance to be fooled. A simulation of 2,000 such A/A tests, each with 5,000 users per arm and checks every 500 users, gave:
stopped if significant at any of 10 looks: 19.7% false positives
looked only once, at the end: 4.8% false positivesLooking only at the end gives about the expected 5%. Peeking pushes the false positive rate to nearly 20%. The fix is to fix the sample size in advance and read the result once, or to use a method designed for continuous monitoring, such as sequential testing, which adjusts the thresholds.
07Other things that go wrong
Running many metrics or many variants raises the chance that something looks significant by luck, so correct for multiple comparisons or name one primary metric beforehand. Users who see both versions, or who are not randomised properly, break the assumptions. A test that does not cover a full weekly cycle can miss day-of-week effects. A sample ratio mismatch, where the group sizes differ from the planned split, usually points to a bug in assignment and the results should not be trusted until it is explained.
08Common interview questions
Q: What is a p-value?
A: The probability of seeing a result at least this extreme if the null hypothesis were true. It is not the probability that the null is true.
Q: What are Type I and Type II errors?
A: Type I is a false positive (alpha, commonly 0.05). Type II is a false negative, and power is one minus that rate.
Q: Why decide the sample size beforehand?
A: To keep the error rates as designed and avoid stopping when a result happens to look good.
Q: A result is significant. Do you ship?
A: Check the confidence interval and whether the lift is large enough to matter, and look at guardrail metrics such as revenue or latency.
Q: How many users for a smaller effect?
A: Many more. Required sample size grows roughly with the inverse square of the effect size.
Q: Why randomise?
A: So that the groups are comparable on average and the change is the only systematic difference.