Multivariate Testing

Multivariate testing (MVT) is a type of experimentation method, and in the context of Optimizely, it involves testing multiple variations of multiple elements simultaneously to determine the most effective combination. Unlike A/B testing, which tests variations of a single element, multivariate testing allows you to test various combinations of different elements on a webpage or app.

As an example, if testing a CTA button, below are some options for testing different elements and versions that could then be combined:

1. Element: Size

   • Small

   • Medium

   • Large

2. Element: Background Color

   • Red

   • Orange

   • Blue

3. Element: CTA Text

   • Submit

   • Next

   • Continue

An MVT test enables you to experiment with different combinations of size, background color, and CTA text. It generates a table of these element combinations, allowing you to test and evaluate various user experience variants.

The goal of multivariate testing is to find the most impactful combination of elements that maximizes the desired outcome, such as higher conversion rates or increased user engagement. It is particularly useful when you want to test multiple changes simultaneously and assess their collective impact on user behavior.

When to Use MVT Testing

Multivariate Testing (MVT) is perfect when you want to test multiple elements of a visitor’s experience to find the best combination or identify which element has the most impact.

Advantages

• Uncover Hidden Insights: MVT can reveal unexpected combinations of elements that drive results.
• Faster Results with Partial Factorial Analysis: By using partial factorial analysis, MVT can quickly achieve statistical significance even when testing multiple elements.
• Detailed Reporting: Some platforms provide insights into which elements have the biggest influence on conversion rates.

Disadvantages

• Longer Testing Time with Full Factorial Analysis: If you run a full factorial test, it can create many variations, significantly increasing the time needed to achieve statistical significance—especially when testing several elements with multiple versions.

Full-Factorial vs. Partial-Factorial Testing

Most platforms only support full-factorial testing, which tests every combination of element versions. This approach increases the number of variations and test duration as more elements or versions are added. To avoid long testing times, keep the number of elements and versions in a single MVT test to a minimum.

Partial-factorial testing (e.g., the Taguchi Method) tests only select combinations, using statistical models to predict untested results. However, this method is rarely supported by testing platforms. For more details on Optimizely's capabilities, check their Support Pages. 

Conclusion

While multivariate testing provides more granular insights into the performance of individual elements and their interactions, it often requires a larger sample size and can be more complex to set up compared to simpler A/B testing. However, the insights gained from multivariate testing can be invaluable for making informed decisions about website optimization and marketing strategies.

What happens when your experiment goes live?

• Visitors to your website are randomly assigned to one of the combinations. Each user sees a unique combination of the variations you've created.
• As users interact with your site, Optimizely collects data on how each combination performs in terms of user engagement, conversions, or other goals you've defined.
• Optimizely performs statistical analysis to determine which combination is the most effective. It identifies the winning combination based on user behavior and the defined goals.
• After the test concludes, you can make data-driven decisions about which combination of elements works best. The winning combination can then be implemented on your site.

Multivariate testing is beneficial when you want to understand not only which individual elements contribute to better performance but also how these elements interact with each other in different combinations. It allows for more nuanced insights into the impact of various design choices on user behavior. Keep in mind that multivariate testing may require larger sample sizes compared to A/B testing due to the increased number of combinations being tested.

Learn more about multivariate tests in Optimizely.