People tend to trust statistics, and they have good reason for doing so. When you see a claim backed up by numbers, it’s hard to turn your head at those results. Within your marketing agency, you probably support your services with statistics that show how effective they are. However, if you’ve arrived at conclusions based on the results of A/B testing, your numbers might not be as strong as you think. ThoughA/B testingis incredibly common for marketing agencies, and it is encouraged as a best practice in web marketing, it is not infallible. As you assess whether your results are truly reliable, ask yourself these questions to learn if your A/B test stats are fooling you.
Did you continue tracking results after your hypothesis was confirmed?
Once you see results that confirm a hypothesis, it might be tempting to end the test, because you’ve achieved the results that you want. With this type of bias built in to favor certain results, you may not see any reliable numbers on the other side of the test.
Did you establish a control for your A/B testing?
Any good experiment will utilize a control group, and A/B testing should be no different. If you want to see how a new page layout compares to an existing one, you’ll need to compare these in real time over an extended period.
Were you running multiple tests at once?
Even with a control in the test, you might see skewed results if you try to run too many tests at a given time. Adding too many new variables can muddy the results, causing you to become uncertain of which, if any, changes are producing winning results for your site performance.
Did you have a sufficient sample size?
Sample size can be tough for some sites because traffic is limited. The best thing you can do in this situation is run the test for a longer period to maximize your data and create a clearer interpretation of the results you’re seeing.
Agencies face constant challenges in the world of digital marketing, but the right resources can help your agency stay afloat and thrive. At Pennington Creative, we know the value of havingcontent creators and designers at your disposal—with our help, you can focus on the numbers and analytics while we continue SEO efforts and content creation for your clients.
