How to test SEO changes on Google with split testing

You can't A/B test Google the way you test visitors. Googlebot sees one version of each URL, and showing it something different from what people see is cloaking, which breaks Google's spam policies. SEO split testing works around this by splitting pages instead of visitors: take a set of similar pages (same template, similar traffic), divide them at random into two groups, change one group, and compare its clicks from Google with the unchanged group over the following weeks.

The unchanged group is the point. Search traffic moves with seasons, news and algorithm updates. A plain before-and-after comparison blames or credits your change for all of that. With a control group, both groups go through the same seasons and updates, so the gap that opens between them is the effect of your change.

Why you can't A/B test Googlebot per visitor

A normal A/B test randomly shows each visitor version A or B and keeps them there with a cookie. That doesn't work for search rankings, for three reasons:

• Googlebot "generally doesn't support cookies", so it only sees the version available to browsers without cookies (Google Search Central on website testing).
• Google ranks one URL with one set of content. It can't rank version A for half its users and version B for the other half.
• Showing Googlebot one set of URLs and humans another "is called cloaking, and is against our spam policies". Google defines cloaking as "presenting different content to users and search engines with the intent to manipulate search rankings and mislead users" (spam policies).

User A/B tests on your pages are still fine for measuring conversion, as long as you follow Google's testing rules: use rel="canonical" on alternate URLs, use 302 (temporary) redirects rather than 301s, and "remove all elements of the test as soon as possible" when you're done. They just can't tell you what a change does to search traffic.

Page-group tests vs time-based tests

Time-based tests aren't useless. On a small site they may be all you have. But write down every Google update, holiday and marketing push in the window, and be skeptical of small effects, because normal week-to-week swings in search traffic can produce them on their own.

How to pick control pages

The two groups need to behave alike before the change, or the test can't separate your change from their differences. Large sites have published how they build matched groups.

Pinterest assigns pages by hashing the page URL together with the experiment name, so each page lands in a group at random and several experiments can run at once (Pinterest Engineering). Etsy found that simple random sampling still produced groups that differed in a statistically significant way before the test began, because a few pages get far more traffic than the rest. So it ranked pages by visits, cut them into bands, and assigned pages from every band to every group. It used groups of 1,000 pages and two control groups instead of one, so it could check the controls against each other (Etsy Code as Craft).

For most sites, that translates into five rules:

1. Use one page template only, such as all product pages or all location pages.
2. Drop pages with odd traffic, such as seasonal pages or pages that just went viral.
3. Sort the remaining pages by the last eight weeks of Google clicks, then alternate them into groups (or assign at random within bands), so both groups get a similar mix of big and small pages.
4. Check the groups track each other for four to eight weeks before you change anything. If the ratio between them wobbles a lot, you need more pages or a longer test.
5. Hold the control group completely still. No content edits, no new internal links, no template tweaks.

How to measure the result

Use clicks from Google Search Console for each page, summed by group per week. Rankings are too noisy and don't account for click-through rate changes, and clicks are what you want anyway.

The simplest honest method is a difference-in-differences, which compares the ratio between groups after the change with the ratio before it. Etsy went a step further and used Google's CausalImpact package to build a synthetic control from its two control groups.

Worked example

An online retailer has 400 product pages on the same template. It tests a new title tag format that puts the product type before the brand. After stratified sampling, 200 pages go in each group.

A before-and-after reading of the changed pages says clicks rose 14.9% (13,100 ÷ 11,400). But the control group also grew 5% over the same weeks, with no change at all.

The difference-in-differences estimate adjusts for that. If the change did nothing, the changed group would have grown in line with control: 11,400 × (12,600 ÷ 12,000) = 11,970 clicks a week. It got 13,100, so the estimated effect is 13,100 ÷ 11,970 − 1 = 9.4%.

To judge whether 9.4% is real, look at the weekly ratio. In the six weeks before, it ranged from 0.93 to 0.97. In the four weeks after, it ranged from 1.02 to 1.06, entirely outside the earlier band. That's a clear result. If the post-change weeks had overlapped the old band, you'd keep running or call it a draw.

How long to run an SEO split test

Pinterest reported that effects usually started to show "as early as a couple of days after launch" and grew for a week or two before becoming steady. It also learned to stop losing tests fast. A test of rendering content with JavaScript showed a traffic drop on day two, the team waited, and after switching it off it took "almost a month" for those pages to recover (Pinterest Engineering).

A practical plan is four to six weeks of test data after the change, with a check at the end of week one to stop anything that is clearly hurting traffic. See how long to run an A/B test for the general principles.

What to test

Changes that apply across a template are the natural fit:

• Title tag formats and length
• Meta descriptions, which can change the snippet people see and the share who click
• H1 wording
• Intro copy above the fold
• Internal links between related pages
• Structured data, such as product or FAQ markup
• Page layout changes that move or hide text

Published results show why testing beats best-practice lists. At Etsy, even subtle title tag changes produced "large and statistically significant changes in traffic", some positive and some negative, and shorter titles did better. At Pinterest, making duplicate title tags unique produced no significant change, while better text descriptions on pages led to follow-up work that raised traffic by nearly 30% in a year. Etsy concluded that different strategies are likely to work for different websites.

Outtest runs Google SEO changes this way, as page-group tests that compare changed pages with similar unchanged pages using Search Console data. The same method works for AI search, covered in how to test whether ChatGPT, Claude and Perplexity recommend you.

Step by step

1. Pick one template with at least a few hundred pages.
2. Pull eight weeks of Search Console clicks per page.
3. Remove outliers, then split pages into matched groups.
4. Confirm the groups track each other before the change.
5. Change one group only, and note the date.
6. Check at one week for damage. Stop if the changed group is clearly falling behind.
7. At four to six weeks, calculate the difference-in-differences and check the weekly ratios.
8. If it wins, roll the change out to the control group too and confirm both groups rise. Etsy did this, and the gap between groups disappeared as expected.

Common mistakes

• Testing across templates, so the groups were never alike.
• Changing something site-wide (navigation, speed, redesign) during the test.
• Measuring average ranking instead of clicks.
• Calling a result from a before-and-after chart with no control.
• Leaving a losing test running because "Google needs time", which is how Pinterest ended up waiting almost a month for traffic to recover.