One-Tailed vs Two-Tailed Tests Explained

Use a two-tailed test unless, before any data arrives, you can say that a result in the opposite direction would lead to exactly the same decision as no result at all. The difference between the two comes down to one thing: how you spend your significance level. A two-tailed test splits it between both directions. A one-tailed test bets all of it on one. That bet is why the same data produces a one-tailed p-value exactly half the size of the two-tailed one.

The halving is also the trap. A test that misses significance two-tailed can often be pushed under the line by relabeling it one-tailed after the fact. Once you have seen the data, that relabeling stops being a design choice and becomes p-hacking. So you have to pick the tail when you set up the test, not when you read the result.

A one-tailed test, also called a directional test, asks a single question: is the new version better? (Or worse, if that is the direction you name.) The null hypothesis is still "no difference". The test simply puts its whole 5 percent tolerance for false alarms on one side of the distribution, the region of results so extreme that you would reject the null.

This concentration is what makes it more powerful. With the whole of alpha sitting in one tail, a smaller effect in that direction is enough to reach significance. Georgi Georgiev of Analytics-Toolkit puts the saving at 20 to 60 percent fewer users for the same power. In his conversion example, a variant at 21 percent against a control at 20 percent, with 10,000 users in each arm, comes out at p = 0.039 one-tailed. That is significant at the usual 0.05 line.

The cost is blindness in the other direction. In the UCLA OARC example, the same data that gives p = 0.0001 in the predicted direction gives p = 0.9999 when the test points the other way. A one-tailed test aimed at "improvement" cannot tell you that a new checkout page is losing sales. It can only tell you that it found no improvement.

There is no price in money here. The price is in credibility. Levine and Banas, writing in Communication Monographs in 2002, argue that having a directional hypothesis is necessary to justify a one-tailed test but probably not sufficient. CASRAI says reviewers now treat one-tailed results with skepticism unless they were justified in advance.

Who should use it: a team that wrote down the direction before launch, and that would take the same action whether the variant is worse or merely not better.

A two-tailed test, or nondirectional test, asks whether the versions differ at all, in either direction. At α = 0.05 it sets aside 0.025 for "significantly better" and 0.025 for "significantly worse". Each direction therefore needs stronger evidence than a one-tailed test would ask for.

What it does well is catch surprises. If the new button hurts conversion, a two-tailed test can say so, which is why Mida's guide to A/B testing calls it the correct default. It is also the convention readers expect. Jeff Kaufman published a correction after an online A/B calculator silently gave him a one-tailed p of 0.053. The two-tailed figure for the same data was 0.108, which is clearly not significant.

Where it falls short is power. In Georgiev's example the same 20 versus 21 percent data gives p = 0.078 two-tailed, which misses the 0.05 line. To detect that effect two-tailed, you need more users. Georgiev argues that for most business decisions this extra caution is the wrong tool rather than the rigorous one, because it answers a question nobody acts on.

Who should use it: anyone who has not committed to a direction in writing, and anyone who would act differently on "worse" than on "no difference".

Neither number tells you how likely it is that the new button works. That is a separate misreading, covered in what a p-value tells you. This page only settles which of the two numbers you are entitled to report.

Timing, more than math. Every source here agrees on one point, including Georgiev, the strongest defender of one-tailed tests. The tail must be fixed before the data is inspected. CASRAI is blunt: choosing the tail afterward to halve your p-value is p-hacking. It is also hard to catch. The switch needs no new data and leaves no obvious trace in the write-up. Stefan Faridani's 2026 preprint goes further and argues that some forms of selective reporting cannot be seen in histograms of published results at all. Tail-switching belongs to the wider family of trying analyses until one crosses the line.

What you would do if the variant lost. Georgiev's test is practical. If a worse result and a merely-equal result both mean "keep the old version", then the decision is one-sided, and a one-tailed test fits it. If a worse result would trigger something different, such as an investigation, a rollback, or a warning to another team, then you need the test that can see harm. Mida's argument is the same seen from the risk side: a one-tailed test aimed at improvement can let a harmful change through unnoticed.

Where the disagreement sits. It is not about whether one-tailed tests are valid. Pre-specified, they are. The open question is whether they should be the default. Georgiev says yes for most business tests. Levine and Banas, Mida, MetricGate and Atticus Li say no. Li adds that sample size and test duration usually matter more to the outcome than the choice of tail. None of these sources measures how often real A/B teams make the wrong choice. Until someone measures that, the dispute is about judgment, not evidence.

Default to two-tailed. It is what A/B platforms have used by default, what readers assume, and the only option that will flag a variant that makes things worse.

Choose one-tailed only if three things are true before launch. You have written down which direction you are testing. You would make the same decision on a worse result as on a flat one. And you need the 20 to 60 percent sample saving badly enough to give up seeing harm. If you do choose it, report the two-tailed figure as well.

If the test has already run, the choice is made. Whatever tail you planned is the tail you report. A two-tailed 0.06 stays 0.06.