A/B testing for indie hackers and solo founders

If you run a product on your own, your traffic is probably small and your time is the scarcest thing you have. Most split testing advice assumes tens of thousands of visitors a week. At a few thousand a month, the rules change.

So test big swings one at a time, judge them on revenue, and skip testing altogether when the numbers can't support it.

Why testing is different when you're solo

Small traffic can only see big effects. The fewer visitors you have, the larger a difference must be before it stands out from chance. A 10% improvement is real money, and it's invisible at 3,000 visitors a month.

Your time is the real cost. Setting up a test, waiting six weeks and reading the result is time you're not spending on the product or on getting more traffic. A test has to be worth that.

You can make big changes fast. A solo founder can rewrite the whole homepage or double the price in an afternoon, with no one to ask. That's an advantage for testing, because big changes are the only ones your traffic can measure.

Draws will be common. When a test ends without a clear winner, the original stays. That's a result, not a failure. It tells you the change wasn't big enough to matter at your size.

What your traffic can detect

A made-up example. Your site gets 3,000 visitors a month and 2% pay. With standard settings (95% confidence, 80% power):

To detect a 20% lift you would need 21,109 visitors per version, 42,218 in total. That's 14 months of traffic.

So at this size, a test is worth running only when the change could plausibly move revenue by 60% or more. Doubling a price can do that. A new testimonial can't. Put your own numbers in the minimum detectable effect calculator, and read the low traffic guide for more options.

Big-swing tests worth running

Judge all of these on revenue per visitor, not signups. At small scale, a few free signups can make a losing version look like a winner. See revenue per visitor.

1. Double the price

Test your current price against twice as much. If half as many people buy, revenue is flat; if more than half still buy, you win. See how to A/B test pricing.

2. Remove the free plan

Test your current free plan against a paid-only offer, with or without a trial. Measure revenue per visitor over at least 30 days.

3. Trial versus pay now

If you have a trial, test paying on day one. If you don't, test a 7-day trial. More in free trial vs no trial.

4. A new homepage

Rewrite the homepage around one customer and one problem, and test it against the current one as a whole page. Measure revenue per visitor. The landing page guide covers what to change.

5. Straight to checkout

Send the homepage button straight to checkout for your most popular plan, instead of to a pricing page. Measure revenue per visitor.

When to skip the test

Below about 1,000 visitors a month, even a change that doubles sales takes over two months to confirm at a 2% paid rate. At that size:

• Fix obvious problems without testing: broken links, a slow page, a checkout error.
• Ship the change you believe in, note the date, and compare revenue per visitor for the four weeks before and after. It's weaker evidence, because other things change too, but it's better than a test that never ends.
• Spend the time on traffic. Doubling visitors makes every future test twice as fast.

Common mistakes

• Testing button colors and microcopy at 3,000 visitors a month.
• Calling a winner at 200 visitors per version because one side has four sales and the other has one. Check it in the significance calculator.
• Running two tests at once on a tiny site, halving the traffic for each.
• Judging on signups, where a free plan usually wins.
• Letting a test run for months with no end date.

How Outtest fits

Outtest's Starter plan is $29 a month for 3 tests. It connects read-only to your payment tool (Stripe, Polar, Lemon Squeezy, Gumroad, Dodo Payments, Creem and others) and your analytics, and its Analyst agent looks for where your funnel loses the most money before the Lead agent plans what to test. Tests are judged on revenue per visitor, and the install is one pasted line.

A test needs at least 7 days, a 90% chance of beating the original and a lift of at least 10% to win, and it ends as a draw at day 42. With a few thousand visitors a month, only big changes like the ones above will clear those bars in time. Under about 1,000 visitors a month, most tests will end as draws, and your money is better spent on getting traffic first.