How to measure A/B tests with Stripe revenue

To measure an A/B test with Stripe revenue, you need one link between two systems: which version a visitor saw, and which Stripe payment they made. Assign each visitor an ID and a version when they land. Pass that information into Stripe when they pay, using client_reference_id on Payment Links, Checkout and pricing tables, or metadata on the Checkout Session, subscription or customer. Then add up net revenue per version after refunds and divide by the number of visitors assigned to each version.

That gives you revenue per visitor (RPV), the fairest single number for a pricing, paywall or checkout test. It credits a version that sells fewer but bigger plans, and it penalizes a version whose buyers ask for refunds.

Why judge on Stripe revenue

Signups and clicks are easy to count, but they reward the wrong things. A cheaper plan gets more signups. A louder headline gets more clicks. Neither tells you which version brings in more money. The fix is to read the result from the system that records money. For more on the metric itself, see revenue per visitor.

The data flow in four steps

1. Assign. When a visitor first lands, give them a random ID and a version (A or B), and store both in a first-party cookie or local storage so they keep the same version on return visits.
2. Carry. When they click to pay, attach the ID and version to the Stripe object that will record the payment.
3. Collect. Listen for Stripe events (or query the API) to see each payment, refund and dispute along with the version.
4. Divide. Divide net revenue per version by the visitors assigned to that version.

The Stripe side depends on how you take payments.

Payment Links

Add client_reference_id to the Payment Link URL as a query parameter. Stripe's Payment Links docs say it can use alphanumeric characters, dashes or underscores, up to 200 characters, and that it's sent in the checkout.session.completed webhook after payment. A link for a visitor in version B might end like this:

https://buy.stripe.com/your_link?client_reference_id=v8f3k2_pricingtest_B

The same page gives two cautions. Invalid values are silently dropped, and the payment page keeps working, so a typo means you lose attribution without any error. And because the ID sits in a URL, it must not contain personal data or secrets.

Checkout Sessions you create on your server

If you create Checkout Sessions through the API, you have two places to store the version:

• client_reference_id, a string up to 200 characters, meant for reconciling the session with your own system (Create a Checkout Session).
• metadata, key-value pairs such as ab_test: pricing_sept and ab_version: B.

The catch is that Stripe metadata "doesn't automatically copy to related objects" (Stripe metadata docs). Metadata on the Checkout Session stays on the session. To put it on the objects you'll query later, use the nested parameters:

• subscription_data.metadata saves to the subscription. Stripe then copies subscription metadata onto each invoice the subscription creates, so renewals carry the version.
• payment_intent_data.metadata saves to the PaymentIntent for one-off payments, and Stripe copies PaymentIntent metadata to the charge.

Metadata allows 50 keys, 40 characters per key and 500 per value. Stripe only returns it to requests made with your secret key, so read it on your server, not in the browser.

Pricing tables

Stripe's embeddable pricing table accepts a client-reference-id attribute and passes it to the Checkout Session's client_reference_id. The same rules apply: alphanumeric characters, dashes or underscores, up to 200 characters (pricing table docs). Set it with JavaScript when the page loads, using the visitor's ID and version.

When payment happens later

Plenty of businesses don't take payment in the same session as the test. Someone signs up for a free trial, or a sales call happens a week later. In that case, write the version to the Stripe customer's metadata when you create the customer at signup. Any later subscription or invoice for that customer can then be traced back.

Which Stripe events to count

Stripe's event types cover each part of the calculation:

• checkout.session.completed fires when a Checkout Session completes, and carries client_reference_id and session metadata.
• checkout.session.async_payment_succeeded fires when a delayed payment method finally succeeds. Stripe's Payment Links docs note that some methods, like bank debits, take 2 to 14 days to confirm.
• invoice.paid covers subscription renewals.
• charge.refunded fires on any refund, "including partial refunds".
• charge.dispute.created fires when a customer disputes a charge with their bank.

Set your rules before the test starts. How many days after the test ends will you keep counting refunds? Will you count only first payments, or renewals too? Changing these after you've seen the numbers is a quiet way to pick the answer you wanted.

Worked example

A SaaS tool tests two pricing pages for three weeks. Version A leads with the $29 monthly plan. Version B leads with the $290 annual plan and shows the monthly price underneath. Each version gets 6,000 visitors. Refunds are counted for 14 days after the test ends.

Judged on paying customers, A wins, 1.50% against 1.40%. Judged on net revenue per visitor, B earns 26% more. Two things to check before you call it:

• The gap rests on six extra annual plans. Put the per-visitor revenue for each group into the revenue per visitor calculator to see whether that's bigger than noise. Revenue is spikier than conversion, and a single $290 sale moves each group's RPV by almost 5 cents.
• Annual plans pay 12 months up front. If you'd rather compare like with like, count 12 months of expected revenue for monthly plans too, but decide that before you look.

Revenue tests need more traffic

Revenue per visitor varies far more than a yes-or-no conversion, because most visitors spend $0 and a few spend a lot. In an example from Kohavi and colleagues, a store where 5% of visitors buy and spend about $75 averages $3.75 per visitor, with a standard deviation of about $30. Detecting a 5% change in revenue then takes over 409,000 users per version, against under 122,000 for a 5% change in conversion rate.

There are two ways to handle this. First, test changes big enough to move revenue by 10% or more, such as plan order, annual discounts or trial length, rather than button colors. Second, agree a cap for extreme values before the test, for example counting no single customer above the 99th percentile of first payments, so one enterprise deal can't decide the result.

Common mistakes

• Assigning the version after the visitor reaches checkout, so only buyers get a version and the denominator is wrong.
• Counting test-mode payments. Filter on livemode.
• Setting metadata only on the Checkout Session and then querying subscriptions, where it isn't.
• Letting one visitor see both versions because the ID isn't stored.
• Dividing by sessions in one version and by visitors in the other.
• Mixing currencies without converting them first.

Outtest connects to Stripe with read-only access and ties each payment to the test version automatically, so every test is judged on revenue per visitor. If you build this yourself, start with the four steps above and check your sample size before launch, because revenue needs more visitors than a conversion test to reach the same confidence.