How to A/B test on Shopify

Shopify lets you A/B test natively through experiments, a type of rollout under Markets > Rollouts in your admin. An experiment splits eligible visitors between a control and a treatment, 50/50 by default, and can test changes to your main theme, your checkout and accounts configuration, your product catalogs (which carry prices) and your discounts. You need the Grow plan or higher, according to Shopify's rollout types page.

What Shopify doesn't give you in an experiment is revenue. For a theme experiment it reports conversion rate, bounce rate, add-to-cart rate and reached-checkout rate (rollout analytics). A version can win on conversion rate and still lose money if it pushes shoppers to smaller baskets. So run the test in Shopify, then check revenue per session for each group before you keep the winner.

What Shopify lets you test natively

Every rollout is a launch, an event or an experiment. Only the experiment type splits traffic into a control and a treatment and compares them. Here is what an experiment can change, from Shopify's docs:

The defaults are a 50/50 split and an end date 90 days after the start, and you can change both before saving. You can also set a traffic allocation, so the rollout only reaches a percentage of eligible visitors, and the split then applies within that group (rollout analytics).

Where Shopify's built-in testing stops

Shopify's requirements and considerations page lists the limits. The ones that matter most:

• Experiments need the Grow plan or higher. Rollouts in general work from Basic.
• You can't apply rollouts to vintage themes, and you can't change Liquid templates as part of a rollout.
• Experiments apply only to your online store. Headless and custom storefront checkouts aren't supported.
• Analytics only count visitors on your online store, cart and checkout.
• A discount code activated in an experiment can be shared, so it can reach visitors outside the test group.
• Custom UI in the information, shipping and payment steps of checkout needs checkout UI extensions, which are "available only to stores on a Shopify Plus plan" (Shopify.dev).

The practical gaps are revenue and flexibility. Experiment metrics are funnel rates with no revenue, each experiment compares exactly two versions, and theme changes go through theme edits rather than Liquid templates. For copy tests on a single landing page, tests with three or more versions, or revenue as the deciding metric, most stores add a third-party testing app or a script that swaps content in the browser.

How to run a theme experiment in Rollouts

1. Duplicate your live theme and make your change in the copy, or edit the main theme inside the rollout.
2. Go to Markets > Rollouts and create a rollout. Choose the experiment type.
3. Add one change, either an edit to the main theme or a replacement with your duplicated theme.
4. Keep the 50/50 split. Set an end date based on your traffic (see the example below), not the 90-day default.
5. Preview both versions on mobile and desktop before you launch.
6. Launch, then leave it alone. Don't publish other theme changes, start a sale or change prices during the test.
7. When it ends, compare conversion rate from Shopify with revenue per session from your orders, then launch the winner or roll back.

Judge on revenue per session, not conversion rate

Shopify defines conversion rate as the share of online store sessions that lead to an order. It counts a $15 order and a $150 order the same. That's fine for a quick read, but the number you actually bank is revenue per session, which is conversion rate times average order value.

Worked example

A homeware store tests a new product page layout that moves "Add to cart" higher and drops a "complete the set" bundle block lower. The experiment runs for 30 days with a 50/50 split.

Shopify's experiment report would show the new layout ahead, with a 9.5% higher conversion rate. But fewer shoppers see the bundle, so baskets shrink by 12.5%, and revenue per session drops by 4.2%. Keeping the "winner" would cost about $0.056 per session, or roughly $2,700 a month at 48,000 sessions.

To get revenue per session for each group, you need to know which group each order came from. Some testing apps record this for you. Otherwise, export orders for the test period and match them to the group by a cart attribute or order note your theme writes. The revenue per visitor calculator then tells you whether the gap is bigger than noise.

Price and discount tests on Shopify

A catalog change in an experiment publishes that catalog's prices to the treatment group only. That makes true price tests possible without an app, but Shopify flags two side effects:

• Your online store "can display different prices or products than Google or other sales channels" while the experiment runs, and product ads, feeds and third-party apps might not show the same offer (requirements and considerations).
• Pausing a rollout returns to the catalog's current prices. It doesn't restore a snapshot, so edits you make during the test stay.

If you run Google Shopping ads, the first point is a real risk. Google Merchant Center crawls landing pages, compares prices with your product feed, and says mismatches "may lead to an account suspension" (Merchant Center help). Test prices on products that aren't in your feed, or keep the test short and watch Merchant Center diagnostics.

A discount experiment is the lower-risk option. Shopify describes it as a way "to test a discount against the discounts that your store offers already". The control keeps your normal discount behavior. Judge it on revenue per session and margin, because a discount almost always lifts conversion rate. For a fuller method, see how to A/B test pricing.

When to use an app or script instead

Shopify's native experiments suit whole-theme and checkout configuration tests on Grow and above. Use a third-party app or testing script when you need:

• Page or section tests, such as one product page template or a headline.
• Revenue per visitor as the deciding metric.
• Tests on the Basic plan.
• More than two versions at once.

Outtest installs through its Shopify app and judges every test on revenue per visitor from your Shopify orders, with signups and conversion rate shown only for context. Any change it proposes to checkout or pricing waits for your approval.

How long a Shopify test needs

Shopify's 90-day default is a ceiling, not a plan. Work out the length from your traffic. Take a store with 2,000 sessions a day and a 2% conversion rate that wants to detect a 20% relative lift (2.0% to 2.4%) at 95% confidence and 80% power. The rule of thumb from Kohavi and colleagues is n = 16 × p(1 − p) ÷ Δ² per group, so 16 × 0.02 × 0.98 ÷ 0.004² = 19,600 sessions per group. Two groups need 39,200 sessions, about 20 days. Round up to three full weeks so every weekday appears the same number of times. Revenue per session is noisier than conversion rate, so a revenue test usually needs longer. The sample size guide and how long to run a test cover the details.

Common mistakes on Shopify

• Running a test across a sale or a big email send, which changes who visits halfway through.
• Letting a test discount code leak to social media or coupon sites.
• Editing the live theme during a theme experiment.
• Calling a winner on conversion rate when average order value moved the other way.
• Price testing products that are in a Google Shopping feed without checking Merchant Center.