Sequential versus Multivariate A|B Testing: Statistical Validity and Optimal Experimental Design
Overview
This paper examines Fung's critique of sequential A|B testing methodology, arguing that it fails to capture factor interactions. The analysis compares sequential A|B testing with multivariate testing approaches, establishing proper conditions for statistically valid A|B testing and evaluating the relative advantages and disadvantages of each method.
What this coursework covers
- Explanation of why sequential A|B testing is suboptimal due to inability to measure factor interactions
- Conditions required for executing statistically valid A|B testing including metric selection and retesting protocols
- Comparison of multivariate testing advantages over sequential A|B testing with discussion of potential disadvantages
Preview
Student’s Name
Professor’s Name
Course
Date
HBR-A Refresher on A|B Testing
Why does Fung believe sequential A|B testing is “suboptimal”?
Previously, many users of A|B testing, especially managers, thought that running sequential tests would give them varied outcomes and projections. The idea stemmed from the belief that they should not vary two or more factors concurrently. However, statisticians have since debunked this unproven technique. According to Fung, sequential tests are largely suboptimal since one does not measure what happens whenever factors interact (Gallo 1). For instance, sometimes users may, on average, prefer blue, however, they would still like red when combined Arial. As a result, such kind of results are missed in the sequential A|B testing. In essence, more sophisticated tests should be run.
What are the proper conditions needed to execute a statistically valid A|B testing?
The issue with A|B testing is that statisticians are likely to make spurious correlations when looking at multiple metrics at the same time. To conduct proper A|B testing, it is important for the statisticians to outline proper conditions that can lead to valid results. First, they should decide on the metrics to use before executing an experiment then select a few. This is because the more one measures, the more they are likely to witness random fluctuations. Second, companies using A|B testing should do retesting (Gallo 1). This is because the result may be statistically significant but with a huge possibility of being wrong
What are the advantages of Multivariate testing vs.