Why stopping an A/B test early misleads you (the peeking problem)

Peeking means checking an A/B test while it runs and stopping the moment the result looks significant. It's the most common way good teams ship changes that do nothing. A 95% significance threshold promises that a test with no real difference will look like a win only 5% of the time, but that promise assumes you look once, at a sample size you fixed in advance. Look every day and stop at the first significant reading, and the false win rate climbs to 20 to 30% over a typical test.

The fix is simple to state. Decide the sample size before launch and read the result once, or use a method designed for repeated looks. The rest of this guide shows why peeking breaks the maths, a simulation you can reproduce, what published sources found, and the options.

Why checking often breaks the maths

When two versions perform the same, the gap between them doesn't sit still at zero. It drifts up and down as visitors arrive, like a random walk. Each time you check, there's a chance the drift happens to be large enough to cross the significance line. One check gives chance one opportunity. Twenty checks give it twenty.

Evan Miller explains it with a small example in How Not To Run an A/B Test (2010). Suppose you check after 200 and after 500 visitors. If you only count the result at 500, you get a false positive 5% of the time. But if you stop at 200 whenever that early check is significant, every test that was significant at 200 and would have drifted back by 500 becomes a false win. You have added false positives and removed none.

A p-value is only valid for the sample size you planned. Stopping when it looks good is choosing the sample size based on the answer.

A simulation you can reproduce

We simulated 20,000 A/A tests, where both versions are identical, to measure exactly how often peeking "finds" a winner that isn't there. The setup:

1. Both versions convert at 3%.
2. Each day, each version gets 1,000 visitors. Draw each version's daily purchases at random from a binomial distribution (1,000 tries, 3% chance each).
3. Keep running totals for 28 days.
4. At each check, run a two-sided two-proportion z-test on the totals so far and count the test as a false win if p is below 0.05.
5. Repeat 20,000 times and count the share of tests that were ever significant at any check.

Any spreadsheet or a few lines of Python will reproduce it. The results:

Checking once gives the promised 5%. Checking daily for four weeks gives 27.8%, more than five times as many false wins. About half of those false results favor B (13.9% of tests) and half favor A, so if you only act on "B wins", you'd ship a change that does nothing in about one test in seven.

Weekly checks are better but still more than double the error. Every extra look costs something.

What published sources found

The simulation matches what others have reported with different setups:

• Evan Miller's worst case: a test at a 50% conversion rate, checked after every visitor and stopped at the first significant result or at 150 visitors, produced a false positive 26.1% of the time. He also showed that if you peek 10 times, what you think is 1% significance is actually 5% (How Not To Run an A/B Test).
• Optimizely simulated no-difference tests of 5,000 visitors. More than 57% falsely declared a winner or loser at least once when checked after every visitor, 26% when checked every 500 visitors and 20% when checked every 1,000 (Optimizely, 2015).
• David Robinson simulated 20-day tests with about 10,000 impressions a day and no real effect. 5.34% were significant on day 20, but 22.68% dipped below p = 0.05 at some point along the way (Is Bayesian A/B Testing Immune to Peeking? Not Exactly, 2015).
• Kohavi, Deng and Vermeer note in A/B Testing Intuition Busters that an early commercial A/B platform showed near-real-time results and recommended stopping when results turned significant, which inflated false positive rates.

The problem was described for medical trials long before web testing. Evan Miller's article points to Armitage, McPherson and Rowe's 1969 paper "Significance Tests on Accumulating Data".

Bayesian methods don't escape it

Switching to a Bayesian "chance to beat" doesn't remove the problem if you use it as a stopping rule. Running the same 20,000 A/A tests and stopping when B's chance to beat first reached 95% (flat prior, normal approximation):

• Checked once, on day 28: 5.1% declared B the winner
• Checked every day: 23.6%

Robinson found the same with a Bayesian expected-loss rule. Checking daily raised the rate of switching to a worse version from 2.5% to 11.8%. /guides/bayesian-vs-frequentist-ab-testing explains when a Bayesian posterior does stay valid under early stopping (short version: when the prior is a real, well-founded belief).

Extending a test is peeking too

The mirror-image mistake is letting a test run past its planned end because it's "almost significant". Adding days until the result crosses the line gives chance extra attempts in the same way. If the test reaches its planned sample without clearing the bar, it's a draw.

When stopping early is fine

Stopping to protect the business is fine. If a version breaks checkout, throws errors or is clearly losing revenue, stop it. The peeking problem is about declaring winners early, not about pulling a broken version. A "loser" call made early is still at risk of being noise, but the cost of wrongly stopping a bad-looking version is usually small. You keep the original.

How to fix it

For sequential methods, see Optimizely's description of its sequential engine above, the always valid inference paper by Johari, Pekelis and Walsh, and Evan Miller's Simple Sequential A/B Testing, which you can run by hand. That rule: choose a number of conversions N before the test. Stop and declare B the winner if B's conversions minus A's reach 2√N. Stop with no winner once the two groups' conversions add up to N.

A peeking-safe routine

For most teams, this routine avoids the problem without new software:

1. Before launch, calculate the sample size and round the duration up to whole weeks with the /tools/ab-test-duration-calculator.
2. Write the end date where the team can see it.
3. During the test, check only health: the traffic split (/guides/sample-ratio-mismatch), errors, and whether a version is breaking something.
4. On the end date, read the result once and act on it. A result short of the bar is a draw.
5. After shipping a winner, keep a small holdout on the original to confirm the lift was real.

Outtest's Referee won't call a winner before a test has run at least 7 days, and ends any test that hasn't cleared its bars by day 42 as a draw, so a test can't run indefinitely waiting for a lucky day. After a win, 5% of visitors keep seeing the original, so a lucky call shows up as a winner that earns no more than the version it replaced.

/guides/how-long-to-run-an-ab-test covers how to set the end date in the first place.