Join Michael Duquet for an in-depth discussion in this video Understanding A/B testing, part of Optimizely Fundamentals.
- View Offline
First off, let's talk about the difference between A/B Testing and Multivariate Testing and then move on to how Optimizely allows you to easily set up both. Optimizely defines A/B Testing as the common name for a test that puts one or more variations of a page (or pages) against one another and measures how the two variations perform against one another for specific defined metrics. A/B Tests, also know as 50/50 tests, usually imply one variation and one control. However, that's not necessarily correct. It is still considered an A/B Test if you have more than one variation.
Say one, two, three, four, five variations tested against the control. As long as you're pitting them against the same baseline, it's considered an A/B Test. Also you may see here it says page (or pages). That's right, you can also test a process as opposed to a static content page. That's something we get into a later video. What is Multivariate Testing? Multivariate Testing is a little bit different in that it's an experiment that has multiple variations of specific elements. The experiment is defined by selecting the elements and their variations. The testing platform will then automatically generate the possible testing combinations of these.
Say for example you have 3 separate variations and 3 separate elements. Multivariate Testing in Optimizely will then create 9 testing combinations, meaning that your traffic will be divided 9 separate ways in addition to the baseline. This is creating different experiences based on different elements in different settings. Say you have one CTA that you want to try on different designs, you'll put in your different designs and then the Multivariate Test will try the CTA in each different design. Optimizely allows you to easily set up A/B Tests and Multivariate Tests once you've already created your experiment.
So we'll click on the Create Experiment button. To begin, and you'll be taken to the Experiment panel where you're given the option to select Experiment Type. Experiment Type will allow you to select A/B Test which is where you are choosing variations of a page against each other. And in the Multivariate Test, we are testing changes to multiple sections of a page. If you have one specific hypothesis that you want to test, an A/B Test will usually suit your needs. However, if you want to try different elements and how they fit together, think of it as pieces in a puzzle, trying to find the final solution, then you'd want a Multivariate Test.
Also, keep in mind that Multivariate Tests tend to take longer because the traffic is divided. In this video we've talked about the difference between A/B Testing and Multivariate Testing, and we've also talked about how Optimizely lets you set up both types of experiments easily, and in minutes.