A/B testing with low traffic

A/B testing with low traffic works if you accept that you can only detect big differences. The amount of traffic a test needs falls with the square of the lift you're trying to find, so a test looking for a 50% lift needs 25 times fewer visitors than one looking for a 10% lift. Low-traffic teams should test bold changes, set a high minimum lift (20 to 50%), measure metrics higher up the funnel where there are more events, and use methods that can stop early when a result is obvious.

They should also skip testing when the numbers can't work. If your traffic can't detect anything smaller than a 60% lift within six weeks, a test on button copy will end in noise, and your time is better spent on customer interviews, fixing obvious problems, or getting more traffic.

How much traffic is "low"?

It depends on your conversion rate and the smallest lift you care about, not on a fixed visitor count. The standard rule of thumb, from Kohavi and colleagues' guide to online experiments, gives the visitors needed per version at 95% confidence and 80% power:

n = 16 × p(1 − p) ÷ Δ²

where p is your conversion rate and Δ is the absolute change you want to detect.

Worked example

A small online business gets 3,000 visitors a week, and 2% of them buy. Split 50/50, that's 1,500 visitors per version per week.

For the 10% row: 16 × 0.02 × 0.98 ÷ 0.002² = 0.3136 ÷ 0.000004 = 78,400 per version. Double it for two versions and divide by 3,000 a week, and you get 52 weeks.

This business can't test for 10% lifts. It can test for 30% lifts in about six weeks, and 50% lifts in two. That sets the strategy. Put your own numbers into the minimum detectable effect calculator to see where your line falls, and read minimum detectable effect, explained for the maths behind it.

Make bold changes

A 30 to 50% lift won't come from a new button color. It can come from changes that alter what the visitor decides:

• A different offer, such as a free trial against a money-back guarantee.
• A different price or plan structure (see how to A/B test pricing).
• A new landing page built around a different promise, not a tweaked headline.
• Removing a step, such as asking for a card later or cutting a form from nine fields to three.

Bold changes also teach you more when they lose. A button color test that ends flat tells you nothing. A test of "free trial" against "30-day guarantee" that ends flat tells you your buyers don't care much about that choice.

Raise your minimum lift

Decide before the test what lift would be worth shipping and what your traffic can detect, and set the higher of the two. If your traffic supports 30% in six weeks, tell yourself and your team that a 12% "win" at week three is not a result. Tests on low traffic often show dramatic early swings from a handful of conversions, and stopping on them is the peeking problem at its worst.

If you use Outtest, raise the minimum lift in Settings (it goes up to 30%) so the Referee only calls winners your traffic can support. Tests that haven't cleared the bars by day 42 end as a draw and the original stays.

Test further up the funnel

Events higher in the funnel happen more often, so tests on them finish faster. The same business above has a 12% email signup rate on its landing page. To detect a 20% lift in signups (12% to 14.4%): 16 × 0.12 × 0.88 ÷ 0.024² = 2,933 visitors per version, about two weeks of traffic. The same 20% lift on purchases would take 13 weeks.

The catch is that signups aren't money. A version that doubles signups by promising something vague can lower the share who buy. Use upstream tests to pick between ideas quickly, then confirm the winner on purchases or revenue per visitor over a longer window, or at least check that the purchase rate didn't fall.

Buy traffic to test messages

If organic traffic is thin, ads let you test messages at a scale your site can't. Two ad angles with a few hundred dollars each can reveal big differences in which promise gets more qualified clicks and signups within a week. Meta's built-in A/B test splits the audience so no one sees both versions (Meta's A/B testing overview), and allows tests from 1 to 30 days (best practices). Use the winning angle as the headline of your landing page. The details are in how to A/B test Meta ads.

Use a sequential test

A fixed-sample test makes you wait for the full sample even when one version is far ahead. Sequential tests are designed to be checked as data arrives and to stop early when the gap is large, without inflating false positives.

Evan Miller's simple sequential A/B test is an easy one to run by hand. You choose a total number of conversions N in advance, then count conversions in the treatment (T) and control (C). If T − C reaches 2√N, the treatment wins. If T + C reaches N first, there's no winner. Miller notes it works best with low conversion rates, and that savings are largest when the treatment is much better than expected (roughly 25 to 50% fewer observations), while tests with no real effect can take longer than a fixed test.

For example, with N = 400, the treatment wins once it leads by 2 × √400 = 40 conversions. If the test reaches 400 total conversions without a 40-conversion lead, you stop and call it a draw.

Pool similar pages

If you have 20 product pages that each get 150 visitors a week, testing one page is hopeless. Testing the same change across all 20 (for example, moving reviews above the price on every product page) pools 3,000 visitors a week into one test. Randomize visitors, not pages, and apply the change to the whole template.

When not to A/B test

Skip the test and just ship, or research instead, when:

• The smallest lift you can detect in six weeks is bigger than any change you could realistically make.
• The change fixes something clearly broken, like a checkout that fails on mobile. Fix it. If you want proof afterwards, big effects show up fast. Kohavi and colleagues note that a 20% drop is detectable in about 1/400th of the time needed for a 1% change.
• You're about to change the product or pricing anyway, so the result would be out of date before it finished.

In those cases, talk to ten customers, watch session recordings, and read support tickets. Then test the few bold ideas that come out of it. For where to start, see what to A/B test first.

A low-traffic testing checklist

1. Work out your minimum detectable lift for a six-week test.
2. Only test ideas that could plausibly beat it.
3. Pick the highest-volume metric that still predicts money, and keep purchases as a guardrail.
4. Run one test at a time on each page, with a 50/50 split.
5. Decide the stopping rule before launch, fixed or sequential, and stick to it.
6. Log every result, including draws, so you don't retest the same idea.