What A/B testing with AI can and can't do

AI helps with most of the work around an A/B test: coming up with ideas, ranking them, writing the new versions, building them, and reading your analytics to find where the funnel loses the most people or money. Much of that work used to need a designer, a copywriter and an analyst. AI can draft it in minutes, so you can test more ideas.

What AI shouldn't do is decide the winner. A model's opinion that version B "looks stronger" is not evidence about your customers, and models tend to overestimate how much a change will help. The winner should be called by a rule you set before launch (a minimum sample, a minimum lift, a confidence bar) and plain statistics on real visitor data. The other job for humans is checking claims, because AI-written copy can invent reviews, numbers and offers that you'd be liable for.

Where AI helps

Ideas and ranking

Recent research gives a fair picture of AI's forecasting skill. A study published in Nature in 2026 built an archive of 70 preregistered survey experiments, with 469 effects and 119,330 participants. Predictions from GPT-4 "were strongly correlated with actual treatment effects, achieving accuracy similar to pooled human forecasts", even for studies published after the model's training cutoff. But the predictions "systematically overestimated effect sizes" (Ashokkumar, Hewitt, Ghezae and Willer, Nature).

So AI is a reasonable way to decide which of 30 ideas to test first. It's a poor way to estimate how much a winner will earn. Those experiments were surveys, not live websites, so expect your own hit rate to vary.

Writing and building variations

AI is fast at this. A good test needs a version that's genuinely different, and AI can draft five distinct angles for a headline or rebuild a pricing page layout in minutes. The limit is judgment about what's true and on-brand, which stays with you.

Spotting leaks

Given access to your analytics and payment data, an AI can compare steps of the funnel and point out, for example, that mobile visitors abandon the checkout's shipping step at twice the desktop rate. That's useful for choosing what to test. Always ask for the underlying numbers and check them, because a fluent summary can still be wrong.

Why AI shouldn't call the winner

There are three reasons, and each has evidence behind it.

First, nobody predicts test results well, including experts. In Harvard Business Review, Kohavi and Thomke report that at Google and Bing "only about 10% to 20% of experiments generate positive results", and at Microsoft as a whole a third of experiments are positive, a third neutral and a third negative. If experienced product teams guess wrong that often, an AI's reading of two page designs is no substitute for data.

Second, models overestimate effects, as the Nature study found. Ask a model to interpret a noisy early lead and that bias can push it toward calling a win too soon.

Third, models can state false things confidently. OpenAI describes hallucinations as "plausible but false statements generated by language models" and argues they happen partly because training and evaluation "reward guessing over acknowledging uncertainty" (OpenAI). A test verdict is exactly where you want "not sure yet" to be a normal answer.

Plain statistics solve this. A rule like "at least 7 days, at least a 90% chance of beating the original, and at least a 10% lift" gives the same answer every time you run it on the same data, and anyone can check the arithmetic. Outtest splits the work this way. Its Builder agent writes each new version and never invents claims, discounts or reviews, while its Referee calls winners with plain maths on revenue per visitor rather than an AI opinion.

Worked example

A founder tests a new landing page headline. After one week:

B is ahead by 14%. It's tempting, for a person or an AI assistant, to call it. The statistics disagree. The standard error of the difference is √(0.03 × 0.97 ÷ 5,000 + 0.0342 × 0.9658 ÷ 5,000) = 0.0035. The gap is 0.0042, so z = 1.19, which means about an 88% chance B is really better and a two-sided p-value of 0.23. That clears neither a 95% bar nor a 90% one.

The founder keeps the test running to its planned four weeks. The final numbers are 600 signups from 20,000 visitors for A (3.00%) and 628 from 20,000 for B (3.14%). The lead has shrunk to 4.7%, and the chance B is better has fallen to 79%. The week-one "win" was mostly noise, which is the pattern described in the peeking problem. A preset rule would have kept the original, and it would have been right to.

Risks to watch

• Invented claims. AI copy can add testimonials, star ratings, customer counts or statistics that don't exist. In the US, the FTC's final rule of August 2024 bans fake reviews and testimonials, including "AI-generated fake reviews" (FTC). In the UK, fake reviews were banned from 6 April 2025 (GOV.UK).
• Offers you don't make, such as a discount, a free trial or a guarantee that doesn't exist.
• Too many versions. AI makes it easy to write eight headlines, but each extra version splits your traffic and raises the smallest lift you can detect (see minimum detectable effect).
• Judging on clicks. An AI asked to "improve conversion" can write copy that wins clicks and loses sales. Judge on revenue per visitor.
• Unreviewed changes to checkout, pricing or cancellation flows, where a mistake costs money directly.

A setup that uses AI well

1. Let AI read your funnel data and propose where to test, with the numbers attached.
2. Let it rank ideas, then pick the bold ones your traffic can actually detect.
3. Let it write and build the versions. A person checks every claim, offer and number.
4. Set the decision rule before launch: sample size, test length, minimum lift and confidence bar.
5. Let statistics call the result, and don't stop early.
6. Keep a holdout group after shipping winners, to check what they earned.
7. Require human approval for changes to checkout, pricing and cancellation, even on autopilot.

What about bandits?

Multi-armed bandits are algorithms that shift traffic toward whichever version is leading while the test runs. They make sense for short-lived content, such as a headline for a one-week sale, where earning more during the test matters more than learning. The tradeoff is that the lagging version gets less traffic, so you learn less about how big the difference really is. For decisions you'll live with for months, like pricing or a new page, a fixed split and a preset rule give a cleaner answer.