A/B testing for Shopify stores

Every order in a Shopify store passes through the same few templates: one product page template, one cart and one checkout. Change the template and every product gets the change, which gives a test far more traffic than any single page could.

So the answer is to test templates and store-wide settings first, and to judge every test on revenue per visitor.

Why testing is different for Shopify stores

Order value moves too, not only conversion rate. A "buy two, save 10%" offer can lower the share of visitors who buy and still raise revenue, or the reverse. If you judge on conversion rate alone you will ship some changes that cost money. Revenue per visitor captures both.

Traffic is spread across many products. A store with 200 products might get 80% of its traffic on 10 of them. Testing one product page limits you to that page's visitors. Testing the template uses all of them.

Seasonality is strong. A test that runs across a sale week and a normal week compares two different kinds of shopper. Keep tests out of big sale periods, or run them entirely inside one.

Shopify also has a native option. Its Rollouts feature can run experiments comparing a new theme or checkout setup against the live one, available on the Grow plan or higher (Shopify Help Center). For a store-specific walkthrough, see A/B testing on Shopify.

The tests to run first

1. The free shipping threshold

Test your current threshold against one a little above your average order, for example $50 against $65 if your average order is $55. Measure revenue per visitor, then check the margin yourself, since shipping costs come out of it.

2. Reviews next to the price

If you already collect real reviews, move the star rating and review count from the bottom of the product page to just under the title. Measure revenue per visitor. Only test this with genuine reviews.

3. Lifestyle photo first

Swap the first product image from a plain product shot to a photo of the product in use, across the whole product template. Measure revenue per visitor.

4. Cart drawer versus cart page

Make "Add to cart" open a slide-out cart instead of sending shoppers to the cart page, or the reverse. Measure revenue per visitor and items per order.

5. A bundle offer on the product page

Add "Buy 2, save 10%" or a matching product under the add-to-cart button. Measure revenue per visitor. This is a pricing change, so check the margin on the winning version.

6. Default collection sort

Sort collection pages by best sellers instead of newest or manual order. Measure revenue per visitor from collection page visitors.

7. Homepage hero

Replace a brand message with your best-selling product and one line on what it does. Measure revenue per visitor from homepage visitors.

How much traffic you need

A made-up example to show the maths. Your store gets 90,000 visitors a month, about 3,000 a day. 2% of visitors order, and the average order is $60, with individual orders usually falling between about $25 and $95.

Because order values vary, a test on revenue per visitor needs more traffic than a test on conversion rate. With standard settings (95% confidence, 80% power):

The revenue test needs about 27% more visitors than the conversion rate test for the same lift, and it is still the right one to run, because it is the number that pays you. A 10% lift takes over two months at this traffic, so aim for changes that could move revenue by 15% or more.

If you tested the same change on a single product page with 10,000 visitors a month, the 15% test would need over nine months. That is the case for testing templates. Compare your own results in the revenue per visitor calculator.

Common mistakes

• Judging on add-to-cart rate. Carts are not orders.
• Letting a test run through a sale. Shoppers in a sale week behave differently from the rest of the year.
• Shipping a discount test without checking margin. More revenue at a lower margin can mean less profit.
• Adding urgency timers or stock counts that aren't true. They cost trust over months, which a four-week test never sees.
• Testing one low-traffic product page when the template covers them all.
• Trusting a lopsided split. If one version got far more visitors than planned, run the numbers through the SRM checker.

If your store is small, the low traffic guide covers which tests are still worth running.

How Outtest fits

Outtest installs through the Shopify app, connects read-only to your Shopify orders and analytics, finds the pages where the store loses the most money, and builds and runs tests like the ones above. Every test is judged on revenue per visitor from your Shopify orders.

Its Builder agent writes each new version and never invents claims, discounts or reviews, so a test of review placement only uses the reviews you have. Changes to pricing and checkout wait for your approval. A version ships after at least 7 days, a 90% chance of beating the original and a lift of at least 10% (defaults you can change), or the test ends as a draw at day 42. Outtest can also test Meta ad copy that sends traffic to the store, covered in Meta ads A/B testing.