A/B Testing: Some Important Rules to Follow
Mar 01, 2016
Simple, efficient, and fast, A/B testing – also known as ‘split testing’ and ‘bucket testing’ – allows you to compare a baseline control sample with a variety of single variable experiments to improve conversion rates. A classic direct mail tactic, this method of testing is easy to execute and is a great starting point for testing copy, layouts, images, and colors.
Think of A/B Testing as a match between “The Champ” and “The Contender” with you setting up the rules, selecting the arena, then letting the data determine the winner.
Contrary to popular belief, you don’t need fancy software or a background in advanced statistics – you just need some common sense and to follow these basic rules.
1. A/B Test Hypothesis
A hypothesis is a prediction you create prior to running an experiment. It states clearly what is being changed, what you believe the outcome will be, and why you think that’s the case. Running the experiment will either prove or disprove your hypothesis. (Source: Downloaded from https://blog.optimizely.com/2015/01/29/why-an-experiment-without-a-hypothesis-is-dead-on-arrival/ on February 26, 2016.
Without a hypothesis, you lack a reason to test – so invest a little time and you can create one like this: “A client testimonial on the landing page will increase conversion rates.” Now you have something to test!
2. Keep it Simple- Limit A/B Test to One Variable
A/B Testing is designed to test one variable – which means you make the one change and the rests remains the same.
Unfortunately this is where so many “tests” go off the rails – they try to test too much and the end result is unclear because you aren’t sure which change drove the result. Keep it simple.
3. Know How to Measure ‘Success’
In this game, there is only one way to win – make sure that it is well defined and accepted by all involved in the test. If you’re hypothesis states “increase conversion rates”, make sure that everyone understand what a ‘conversion rate’ is.
Lacking a clear definition of what success is tends to be the second most common mistake in A/B testing so use proper planning and clear communication up front to get agreement.
4. Quantity and Statistical Significance
Statistical significance is a result that is not likely to occur randomly, but rather is likely to be attributable to a specific cause. (Source: Downloaded from http://www.investopedia.com/terms/s/statistical-significance.asp on February 26, 2016.)
For your A/B test to be successful, you need a large enough quantity.
FREE CALCULATORS: Check out these free tools that will help you get the answers to the above.
5. Test Group and Splits
Back in my early days of direct marketing, we would regularly mail 100,000 people at one time so we would send the control package to [ex] 90,000 in order to achieve the sales goals for the campaign and then we would run tests on the remaining 10,000. And in order to make sure that the test groups were large enough to provide statistically significant results, we would test with groups no smaller than 5,000.
Once the results were in,we ran the math in order to determine [a] if the tests beat control in terms of responses, and [b] if the result was at a confidence level (a measure of the reliability of a result) was acceptable – and we typically wanted to see a confidence level of 95% or higher.
6. Randomization
Bias is something you want to minimize in your test and one of the most common ways this can occur is with the selection process – who receives control and who receives the test(s).
For example, when were mailing 100,000 people, someone might suggest that the control package (the Champ) be sent to the best customers on the list based on dollars spent in the past 12 months so that we had a greater likelihood of hitting the revenue goal for the mailing.
That’s bias.
Instead, we would randomly select who got what – control or test(s).
Since you’re only testing one variable, you want to eliminate the variables in the audience selection process. Your control and test groups should be picked randomly.
You can do this is MS Excel .
Another method of randomization is using the nth name skip method where you select names from your file on for example a basis of selecting every 9th name in the list to test. You will still select the number of names that would be statistically significant to test, but you would select every 9th name. Based on the size of the list and the quantity you wish to test, you might do a larger or smaller nth name skip selection.
7. Test Constantly, Test for ways to Improve Performance
You should be testing as often as you can – but you need to be testing for ways to significantly improve performance. And some things will never significantly improve performance so don’t bother testing.
8. Record your work
Keep a record of what you tested, why, how and the results so that you don’t find yourself repeating tests that don’t need to be repeated – and so that the organization has a record of what you did if you were to leave. (It’s the old ‘if I get hit by a bus’ scenario – your records will help the those that follow you continue moving forward rather than unknowingly repeating what has already been done.)
Patrick McGraw is VP of Higher Educaton Marketing Services and has more than 25 years experience in market research, competitive intelligence, business intelligence including database marketing and CRM, strategic planning, brand development and management as well as operations/campaign management. His work has consistently helped his clients and employers develop and implement more efficient ways to attract and retain profitable customers, enter new markets and launch new products. His areas of focus include the education, hospitality, travel and tourism, hi-tech, telecommunications, financial services, and retail industries on both the agency and customer sides.
Add Pingback