A/B and Multivariate Testing in Sitecore
When first exploring Sitecore, it can be tempting to dive in and begin implementing every functionality the platform has to offer. However, before exploring a more complex feature like personalization, I recommend testing the waters with A/B and multivariate testing.
Simply put, A/B testing tests the efficacy of one variable, such as two different lines of copy or two different images, while multivariate testing allows you to test multiple components at once. These tests are astonishingly easy to set up, and the benefits are numerous: By analyzing the results of these tests, you’re able to optimize user experience and determine what’s working – and not working – within your content.
I think about experience testing through three different approaches: content testing, component testing and page testing.
1. Content Testing
Content testing is the most basic form of testing available within Sitecore and involves testing different content, such as text or images, within a component.
For instance, the headline, description and call to action differ in Versions A and B of this ad.
2. Component Testing
Component testing is similar to content testing, but it tests the use of different components. The content within the component may also be changed:
3. Page Testing
Page testing tests completely different versions of a complete page:
Bear in mind that not all pages will have enough traffic to perform testing. When starting a test, Sitecore shows the number of visitors needed to achieve statistical significance in the results. If historical visitor data is available for the page, Sitecore will forecast the number of days needed to complete the test.
How to Set up Testing in Sitecore
Content and component tests are, in most cases, quick and easy to set up using the “Test the Component” button in the Experience Editor to open the test interface. Page tests are set up from the “Experience Optimization” interface in Sitecore.
How to Configure a Test in Sitecore
Understanding the “why” behind A/B and multivariate testing is one thing, but making configurations to these tests that meet your specific business needs is quite another. However, I think you’ll be pleasantly surprised – dare I say amazed? – by how easy it is to set up configurations for these tests within Sitecore.
Expected Effect of Changes
The first configuration Sitecore requests is the expected effect of changes. This has no impact on the test itself, but rather will track the tester’s ability to predict the outcome. You could even say this adds a little gamification to your experience testing program.
When setting up a test using the Experience Optimization interface, Sitecore will present the variables available to test and will allow you to enable or disable each variable. A simple A/B test focuses on one variable, with more variables it becomes a multivariate test.
The number of variables to be tested and the historical traffic to the page will enable Sitecore to calculate the timing needed to reach statistical significance. The more variables you include in the test, the more traffic you will need to reach significance.
Percentage of Visitors Exposed
Percentage of visitors exposed to test: The percentage of visitors that you want exposed to the test.
For example, if you set the percentage at 60 percent, then 60 percent of visitors will see one of the variations that you have created. The original variation is part of these. The remaining visitors – 40 percent in the above example – will see the original variation and will not be part of the test.
Select the statistical confidence level the test needs to reach before declaring a winner: 90 percent, 95 percent or 99 percent.
For this configuration, 95 percent is the default setting. The higher the level, the longer the test needs to run.
Set up the Test Objective
You have two options to choose from for this configuration:
- Completion of a specific EVS goal (such as signing up for a newsletter)
- Trailing Value/Visit (the engagement value based on page views occurring after the visitors encountered the page being tested, divided by the number of site visits).
To pick a winner, choose one of the following options:
- Automatically select a winner based on the test objective
- Automatically select a winner based on the test objective, unless it significantly decreases engagement value
- Manually select winner
This configuration determines the minimum and maximum time the test will run. At minimum, you can select three, seven or fourteen days. At maximum, you can select fourteen, thirty or ninety days.
That’s it! You’re test is ready to go!
As I hope I’ve demonstrated, setting up A/B and multivariate testing in Sitecore isn’t just extremely useful – it’s also easy. This is a low-risk exercise that can result in valuable insights about your audience and your content.
In short, get comfortable with the test configurations, observe, iterate … and repeat.
Tim Lemke, VP of Creative Technology, Barefoot Proximity