B2B SEO: A/B Testing for Search

The rise of A/B Testing for SEO

Consumer-focused companies have typically been ahead of the curve when it comes to search engine optimization, including SEO split testing. The practice was pulled further into the spotlight by Pinterest when growth engineer Julie Ahn shared a famous post on their successful SEO A/B testing framework. According to Julie, SEO is one of Pinterest’s biggest growth drivers, but in the past it had been difficult to manipulate it for predictable growth.

“You might have a good traffic day or a bad traffic day and not know what really triggered it,” she writes, “which often makes people think of SEO as magic rather than engineering.”

So Pinterest created an A/B testing framework to turn that magic into a “deterministic science.” This testing unearthed important clues about what users wanted. For example, one SEO test involved putting more text descriptions alongside images, and the positive result led to a series of follow-up experiments that resulted in a nearly 30% increase in traffic in 2014.

See? SEO is nothing to scoff at. Small changes can create large and lasting effects.

Zapier: SEO split-testing at a B2B SaaS company

It’s easy to see how Pinterest, with over 175 million active users, can grow rapidly with a few well-executed SEO optimizations. But how does this translate to a B2B context?

Let’s take a look at how Zapier relies on SEO.

Zapier builds automated workflows—a “glue” to connect 750+ web apps and take repetitive tasks off your hands. If you use Mailchimp and Trello, you can connect them to move data from one place to another automatically. Or connect Trello with Calendly, Calendly with Slack, etc.

Zapier’s site is, therefore, made of many, many webpages, each dedicated to individual apps and unique two-app combinations—over 500,000 pages, in fact.

Imagine that someone who uses Trello and Calendly searches for a way to connect the two. Zapier wants their dedicated landing page for Trello and Calendly to be the top result.

Clicking on that result leads to this page with instructions and tools.

Multiply this by endless app combinations. Organic search has been a long-lasting, cost-effective, critical growth channel for Zapier from the very start, because ranking high can be free with the right SEO efforts.

At first, Zapier performed really well in search, but as more competitors entered the market, growth began to plateau. They wanted to start experimenting with more SEO tactics, and as a data-driven company, they wanted to A/B test them before rolling out permanent changes. Instead of building an internal tool to support experimentation like Pinterest did, they came to work with my company, RankScience, where we’ve performed around 20 split tests over a year of collaboration.

Asad Zulfahri, a growth marketer at Zapier who is dedicated to SEO, explains why search remains such a core channel to this day. “With proper SEO, you don’t need as much money to expand rapidly; you don’t need to pay for PPC to grow. It’s one of the key elements of making Zapier profitable as a company,” he says. “For SEO, our goal is to more than double our growth.”

Before we show you some of Asad’s tips and tactics, here’s a quick primer on how SEO A/B testing works and how you can try it yourself.

The difference between A/B testing for SEO versus CRO

As we mentioned earlier, A/B testing for CRO hopes to get more people to convert while they’re on the page, while A/B testing for SEO hopes to drive more traffic to your page in the first place.

The types of changes you make for SEO should appeal to users in Google’s search results, as well as the Googlebot that determines your pages’ search rankings.

These two modes of testing involve one main structural difference. To split test for CRO, you split your site visitors (humans!) into two groups. Using a tool like Optimizely, VWO, Google Optimize, or your internal tool, you take all incoming visitor traffic and show half of them your original page, while you show the other half a variant that the tool imposes.

But to split test for SEO, we divide your webpages—not the audience—into two groups. We take a stack of the same type of page (such as all your /blog or /product pages) and implement the SEO change in one group but not the other.

We do this because we can’t simply duplicate one original webpage and send half of Google’s traffic to one version of your page and half to the other. Google doesn’t like seeing large amounts of duplicate content. And you can’t show different information to users than you show to Google— this could be considered cloaking (read more here). Not only that, but your original page might be much older and more established than your new variant, and therefore have better SEO ranking, so the comparison wouldn’t be anywhere near apples to apples.

Because of the way A/B testing for SEO is structured by splitting groups of test pages, it is better suited for large sites with many of the same page, to control for pre-existing, individual page variations within each group.

There is no hard-and-fast rule for how many pages you need. At RankScience, right now we’re recommending that you try this for groups of at least 100 similar pages (50 in each group). These might be 100 blog posts, 100 product pages, 100 review pages, 100 location pages, etc. (In Pinterest’s case, they were testing “board pages,” and in Zapier’s case, they’re testing product pages.) But of course, the number of pages required is always case by case. It depends on the site and existing search traffic—although 100 is a good rule of thumb.

What should I be measuring?

Keep an eye on three metrics:

1. Total number of clicks (total number of site visitors from the SERP) - How many people clicked on your search result?
2. Total impressions - How many people are seeing your page on the SERP? (You’ll also want to look at click-through rates—if you’re getting more impressions but not more clicks, you may need to adjust your titles and meta tags to make them more enticing.)
3. Average SERP ranking - This is the average ranking of the pages in one group on Google.

How many tests can I do at once?

Only one A/B test at a time should be done on a given set of pages, to isolate the adjustment. So if you’ve only got one stack of pages to split-test, make sure to sequence tests one after the other.

But you can run two or more tests at the same time on different areas (groups of pages) of your website if you segment pages carefully. At Zapier, the URL structure of the site lends itself well to segmentation because we can test all pages that have the same URL path. For instance, it would be easy to segment out and A/B test only and all pages that are:

• Zapbook pages with one service: zapier.com/zapbook/(service1)
• Zapbook pages with two services. zapier.com/apps/(service1)/integrations/(service2)
• Zap Template pages (a Zap Template is a blueprint for a repeated task): zapier.com/zapbook/zaps/(number)/(zap)

For how long should I run an A/B test?

At RankScience, we run experiments for 21 to 28 days to get the full dataset, but there is no hard rule because sites have such different volumes of search traffic. Traffic is the fuel for A/B tests. The more fuel you have at your disposal, the faster you can conclude experiments because they’ll reach statistical significance.

This is why it’s quicker to A/B test changes to website pages that have already been up and running for a while. They’re ranked on Google already, and you’ve measured steady traffic coming their way. Contrast this to A/B testing brand new pages that are still being developed and launched: they’ll need time to attract significant, stable traffic.

In Pinterest’s case—consider them a high-traffic site—they saw initial results from a change after a just couple of days. Then the variant group would creep further away from the control group until it stabilized after one or two weeks. In one test rendering with Javascript, it took them just two days to see a disastrous dip. They turned the experiment off after four days, and it started to tick back up on day six.

Tools to help you

There are three paths you can take when you try SEO split-testing. We’ll show you the DIY version, then we’ll talk about building or buying experimentation tools that make the process more accurate and hands-free.

DIY

If you want to split-test without the help of testing software, you can manually group the pages yourself and watch their traffic numbers change in Google Search Console.

1. Divide site pages into two groups: control and variant.
2. Make a change to the variant group.
3. Wait 21–28 days while monitoring clicks, impressions, and CTR with Search Console.
4. Collect results: Go into Search Console and filter for your control group pages. Determine their average CTR during the time period since you made the SEO change by filtering for the last 21–28 days before and after you made the change (Search Console lets you compare date ranges). Now, you have the before-and-after average CTR for your control group.
5. Do the same for your variant group pages.
6. Compare the variant and control CTRs and determine the largest improvement. The pages with the highest CTR improvements are the winners, and you should be able to determine whether the SEO change was beneficial or not!

Here’s the stats page in Google Search Console, filtered for one group of pages. Check the boxes for Clicks, Impressions, and CTR, and filter to compare the date ranges before and after the test.

---

So, what should I test?

There’s one important truth about SEO: There’s no one list of changes I could recommend that you make to your site. It’s really not one-size-fits-all. The test ideas are endless. Some unique, unassuming tactics may end up working like a charm for you, while others will actually hurt your traffic. (This is why A/B testing is so helpful.)

As you start searching for test ideas, you’ll see hundreds of SEO site audit guides on Google—see this technical SEO checklist, for example. While these lists are a great starting point, we recommend digging deeper. Anything can be worth a try!

Start by reflecting on high-level questions such as:

My first three things to test

Alright, even though I insist SEO isn’t one-size-fits-all, I do have a “first three” checklist that fixes common mistakes many B2B SaaS companies make.

1. Title Tags

Best practice: Put important keywords first in title tags.

Title tags become a page’s clickable headline in search engine results, and are critically important for usability, SEO, and social sharing.

I often see B2B SaaS folks put their company name at the beginning of a title tag, which typically isn’t the best choice for SEO. If you’ve done this, switch them around.

• Common mistake: [company name] - [keyword-filled description]
• Better: [keyword-filled description] - [company name]

In the example above, Zapier has it the better way:

• Common mistake: [Zapbook - Zapier] - [Calendly Integrations]
• Better: [Calendly Integrations] - [Zapbook - Zapier]

---

2. Meta Descriptions

Best practice: Make your meta descriptions enticing to click on!

Meta descriptions are the descriptive texts in search engine results, and together with title tags they contribute to the first impression you make on a potential customer.

I see a lot of B2B SaaS companies writing metas with fluffy branding language. Instead, focus your meta descriptions around your unique value propositions (UVPs): what a user can get from you.

In the example above, Zapier highlights Calendly’s value of eliminating email back and forth and helping businesses save time. You can test a few different UVPs to see what sticks.

Make your metas count!

---

3. H1 Headers

The H1 header appears on a webpage after you click a search result. This is typically the most-emphasized, highest-level title or heading for the page, like a headline for a newspaper article. On the RankScience homepage, ours says “Grow your website traffic with A/B testing.”

Make sure you have important keywords in it. I often see companies sacrifice H1 headers for the sake of site design, either removing them or having non-optimized text. But we recommend that H1 tags be built into the CSS of the page using best practices.

---

What else can I test?

Once you’ve tried these fundamental SEO changes, get creative as you come up with your own.

Here are just a few areas of a page that Zapier tests.

Asad’s favorite advice for finding great ideas? Talk to customers. “SEO works so hard to please the Google bot, but in reality, there are actual people using your site,” he says. “So my tip is to talk to customers, get their stories, and then use their actual language. That’s the most effective content to engage them.” And if he is browsing another company’s site and sees a feature he likes, he might test that too.

---

Test the unexpected

You can test just about anything—even small UX changes!

RankScience co-founder Dillon Forest was mapping out some potential changes and experiments for Zapier when he noticed that some of their site text was small and hard to read. If he was having a hard time seeing it, he figured potential site visitors may have the same issue. Here was the text at 12-point font.

He made the font a little bigger, 16 point, and A/B tested it.

This actually turned into an SEO win! After 28 days, we saw increased impressions and clicks to those pages. Our theory? The increased font size led to more clicks, which led to a higher SERP position, which drove more traffic to the site. The win has continued to stand to this day.

So far, we’ve talked about what SEO A/B testing is, the tools you need to do it, and a few tests you can try. There’s one last thing to consider that might come up as you start doing more SEO testing at your company: the need to balance it with branding.

---

Balancing branding and SEO

Every now and then, I see a tug-of-war between a company’s branding choices and their SEO changes, where one compromises the other.

Here’s an example. Asad once added the word “sync” to URL meta description tags, as in “sync Slack with Trello.” He was excited to see a strong, positive test result and brought it to the rest of the marketing team, but learned that “sync” is actually against Zapier brand guidelines.

Technically, Zapier is a service that moves data to other services, it but doesn’t exactly sync the two. So they couldn’t use that idea, even though it provided a temporary uplift on click-through rate. (And anyway, users would probably feel misled when they eventually figure out there’s no real syncing. That would hurt Zapier’s brand and ranking in the long run.)

Here’s a different example from an early title tag test for the Zapier app directory pages.

The Zapier app directory is called the “Zapbook,” a branded word, and their branding tagline is “Better Together.” Zapier used these two phrases in their title tags.

For instance, in a landing page for the integration of Trello (service1) with Magento (service2), the title tag read:

• Original title: Magento + Trello: Better Together - Zapbook - Zapier
• Sitewide convention: https://zapier.com/zapbook/(service1)/(service2) (service1) + (service2): Better Together - Zapbook - Zapier

But we wondered whether those phrases carried enough meaning for users. While “Zapbook” was on brand, we saw it as a weak keyword taking up valuable real estate. “Better Together” makes sense from a branding perspective, but provides little insight into what Zapier actually does. We tried a new title tag that focused more on utility.

• New title: Connect Magento to Trello Integration (Tutorial)
• Sitewide convention: Connect (service1) to (service2) Integration (Tutorial) - Zapier

We chose this title because “integration” is a strong descriptive keyword. “Tutorial” was an experiment—we often test descriptive keywords like “tutorial,” “examples,” and “lessons” when appropriate.

This showed a 5% uplift in CTR from the original.

So we continued to iterate. Search Console showed us that a lot of site visitors were searching with the words “how to connect,” so we tried to match their natural language for our next test.

• Newer title: How to connect Magento to Trello (Integrations) - Zapier
• Sitewide convention: How to connect (service1) to (service2) (Integrations) - Zapier

This test showed a 17% CTR uplift!

This is all to say that SEO changes sometimes require a deprioritization or shift in branding, and vice versa. We don’t have a clear answer for whether to prioritize branding or SEO—that’s a decision that each company/marketer needs to make—but at least when you’re data-driven about it, you can see what the upside would be. In Zapier’s case, for their title tags, the 17% uplift was worth it, but in the case of “sync,” it wasn’t.

---

Off to the races!

We hope you’re leaving with an understanding of A/B testing for SEO and a few ideas for how to get started. Little efforts in SEO can sometimes yield big results, and experimentation makes the relationship between the two much more measurable.

“The impact of an SEO A/B testing framework at Zapier has been very positive. Just the time saved alone is great, where we don’t have to wait for engineering on our end,” Asad says. “I think that every company should have this kind of resource.”

Give it a shot—we wish you luck!