The Difference Between One-Tailed & Two-Tailed Testing

At eTail West, one wise retailer told this audience, “80% of what you think you know about your site is wrong.” His message was one among many ways digital marketers urged their audiences to test (and test often).

They said this back in 2015, but it still holds true today.

The problem is, marketers often assume testing in itself is enough to negate the impact of personal bias, even as many brazenly ignore the statistics behind running a valid test. The results are several surprisingly common misconceptions that can turn even the most well-intentioned tester into a mindless, hypothesis confirming drone.

One of the biggest mistakes a marketer can make is failing to understand the difference between one-tailed and two-tailed tests. And we don’t blame them. Testing vendors don’t necessarily provide the option to calculate statistical significance in more than one way, and if they don’t, they probably aren’t going to bother explaining the difference.

I’m not here to say a one-tailed test is inherently useless, but rather it is a risky point of confusion when understanding the validity of your testing campaigns and making decisions about the user experience on your site or mobile app.

Now let’s get into some detail.

One-tailed tests: What is a one-tailed test?

A one-tailed test allows you to determine if one mean is greater or less than another mean, but not both. A direction must be chosen prior to testing.

In other words, a one-tailed test tells you the effect of a change in one direction and not the other. Think of it this way, if you are trying to decide if you should buy a brand name product or a generic product at your local drugstore, a one-tailed test of the effectiveness of the product would only tell you if the generic product worked better than the brand name. You would have no insight into whether the product was equivalent or worse.

Since the generic product is cheaper, you could see what looks like a minimal impact, but is in fact a negative impact (meaning it doesn’t work very well at all!), but you go ahead and purchase the generic product because it is cheaper.

If this is the case, you’re probably wondering when a one-tailed test should be used? One-tailed tests should be used only when you are not worried about missing an effect in the untested direction.

But how does this impact optimization? If you’re running a test and only using a one-tailed test, you will only see significance if your new variant outperforms the default. There are two outcomes: the new variants wins, or we cannot distinguish it from the default.

Here’s a quick summary:

Two-tailed tests: what is a two-tailed test?

A two-tailed test allows you to determine if two means are different from one another. A direction does not have to be specified prior to testing. In other words, a two-tailed test will take into account the possibility of both a positive and a negative effect.

Let’s head back to the drug store.

If you were doing a two-tailed test of the generic against the brand name product, you would have insight into whether the effectiveness of the product was equivalent or worse than the brand name product. In this instance, you can make a more educated decision because if the generic product is equivalent, you would purchase it because it is cheaper, but if it is far less effective than the brand name product, you’d probably shell out the extra money for the brand name product. You wouldn’t want to waste your money on an ineffective product, would you?

So when should a two-tailed test be used? Two-tailed tests should be used when you are willing to accept any of the following: one mean being greater, lower or similar to the other.

And how does this impact optimization? When running a test, if you are using a two-tailed test you will see significance if your new variant’s mean is different from that of the default. There are three outcomes: the new variant wins, loses or, is similar to the default.

Here’s a quick summary:

Which testing approach should you be using?

Two-tailed!

Two-tailed tests mitigate the risk involved with predicting how future visitors will be impacted by the tested content. By accounting for all possible outcomes, this approach provides more valuable, unbiased insights which can be reported on with confidence.

Testing is supposed to make it easier for marketers to understand the impact of a certain change without the need for IT intervention, but when the difference between one-tailed and two-tailed tests goes ignored, both the marketer’s time and the IT resources are at risk of being wasted. Don’t fall victim to personal bias or winning results which are vacant of real meaning. This testing approach does require more traffic and time, but that is a small price to pay for reliable results.

Now get testing!

Testing plays a huge role in seeing your mobile marketing is making an impact. Find out more about how you can see that it is with: