Someone sends you a test readout: the new checkout flow beat the old one, p = 0.03. Here is what that number says. If the new flow were exactly as good as the old one, and you ran this test over and over, you would see a difference at least this large in about 3 of every 100 runs.

That is the whole claim. It is a statement about data, calculated while assuming there is no real effect. It is not a statement about whether there is a real effect. The formal version, from the Statistics Fundamentals guide to p-values, is that a p-value is "the probability of obtaining a test statistic at least as extreme as the one calculated from your sample data, assuming the null hypothesis (H₀) is true."

Everything that goes wrong with p-values goes wrong at that conditional. People read "probability of the data, given no effect" and hear "probability of no effect, given the data." Those are different numbers, and nothing in the calculation converts one into the other.

What the number is a probability of

The procedure runs in one direction and it helps to walk it in order.

First you write down a null hypothesis: the two versions convert at the same rate, the drug does nothing, the pipeline treats both groups alike. Then you collect data and reduce it to a single test statistic, some measure of how far your result sits from what the null predicts. Then you ask: assuming the null were true, how often would chance alone throw up a statistic this far out or further? That proportion is the p-value.

The philosopher of statistics Deborah Mayo describes the logic as a statistical version of proof by contradiction: you provisionally grant the null, and a very small p-value means the data sit awkwardly with what you granted. Awkwardly, not impossibly. Small p-values happen under true nulls all the time, which is the point of the number.

The 0.05 line is a convention, and a datable one. It traces to R. A. Fisher in 1925. Nothing in probability theory picks it out, and treating it as a border between real and not real is the habit the reform literature has spent a decade trying to break. A graded reading does less damage: below 0.001 is very strong evidence against the null, 0.001 to 0.01 strong, 0.01 to 0.05 moderate, 0.05 to 0.10 weak, above 0.10 close to nothing.

Run enough tests and small p-values arrive on their own

A 5% false positive rate sounds modest until you count how many chances it gets.

Run 20 independent tests at the 0.05 threshold when every null is true, and the chance that at least one comes back significant is 64.2%. Not a rare event. The likeliest outcome.

Aaron Brown, interviewed by Retraction Watch, pushes the arithmetic further. Suppose a field tests 1,000 hypotheses of which only 10% are actually true, using tests with a 5% false positive rate. The 900 false hypotheses produce about 45 significant results. The 100 true ones, given realistic power, produce about 20 that get confirmed. More than twice as many significant findings are wrong as right, and no one has cheated. Brown's point is that journals set a threshold for the p-value but rarely ask for the two things that decide what a passing p-value is worth: the prior plausibility of the hypothesis and the power of the test.

64.2%
Chance of at least one false positive
Across 20 independent tests at p < 0.05 when every null hypothesis is true
45 vs 20
False against true positives
Brown's worked example: 1,000 hypotheses, 10% of them true, 5% false positive rate
0.05
The conventional threshold
Traceable to R. A. Fisher in 1925, not derived from anything

Where the misreading shows up

In A/B test readouts. The most consequential error among people who run experiments for a living is reading the p-value as the chance the variant wins. A practitioner guide in Medium's Better Decisions publication puts it flatly: "A p-value does NOT tell you the probability that your variation B is better than the control A." The same piece warns about peeking, checking results before the test finishes and stopping when the number looks good, which inflates the false positive rate above whatever threshold you thought you set.

In the published literature. A 2026 medRxiv preprint by Choi, Lee, Chavalarias, Shin and Ioannidis scanned more than 22 million PubMed abstracts and 6 million full texts from 1990 to 2025. P-value reporting in full texts rose from 5.2% to 53.3%. The median number of p-values per full-text article went from 2 to 7. Between 94% and 98% of articles that report any p-value report at least one at or below 0.05, and values cluster around 0.05 and 0.001 rather than spreading out the way honest testing would produce. Ten years of reform has not thinned the reliance; exact p-value reporting even stalled, at 51.1% in 2015 and 49.8% by 2025.

In the profession itself. The American Statistical Association's 2016 statement, published on 7 March that year under Ronald Wasserstein and Nicole Lazar, set out six principles and arrived with more than a dozen discussion papers attached, because statisticians did not agree on the remedy. They still do not. A 2019 Wasserstein editorial called for retiring the phrase "statistical significance"; a 2021 ASA task force led by Karen Kafadar clarified that the editorial was not ASA policy. If the experts are still arguing, you are allowed to find the concept slippery.

Five things it is not

The reading What is wrong with it
p = 0.03 means a 3% chance the null is true The p-value is computed by assuming the null is true. It cannot also be the probability of that assumption. The ASA says outright that p-values do not measure the probability that the studied hypothesis is true.
p = 0.03 means a 3% chance the result is due to chance Also ruled out by the ASA statement. The number tells you how often chance alone would produce data this extreme, not how often chance was the cause of the data you have.
p < 0.05 means the effect is big or important "A p-value, or statistical significance, does not measure the size of an effect or the importance of a result." With a large enough sample, a trivial difference clears 0.05 easily.
p = 0.20 means there is no effect It means the data did not supply enough evidence against the null. An underpowered test on a real effect returns large p-values routinely. Absence of evidence is the finding, not absence of effect.
p = 0.05 means a 95% chance of replication Replication depends on the true effect size, the power of the follow-up study and its design. The p-value from one study does not encode any of those.

What to ask when a p-value lands on your desk

Ask for the effect size and the confidence interval, in the units the business cares about. A p-value tells you the result is hard to explain by chance; only the effect size and interval tell you whether it is worth building. A lift of 0.2 percentage points and a lift of 6 points can carry the same p.

Ask how many tests were run. Seven metrics, four segments and three time windows is a lot of chances for 0.05 to be cleared by noise, and the 64.2% figure is what that looks like.

Ask whether the threshold was set before the data came in. Moving alpha afterwards, or stopping a test at a convenient moment, is p-hacking, and it raises the false positive rate above the number on the page.

Ask what you believed before the test. A surprising result from an implausible hypothesis needs a smaller p-value to move you than a confirmation of something you already had reason to expect. That is the whole content of Brown's 45-against-20 arithmetic.

And then report the p-value anyway, as an exact figure rather than a pass or fail, next to the effect size, the interval and the raw difference. The ASA's fifth and sixth principles come down to the same demand: a p-value on its own is not an inference, and full reporting is what makes the inference possible.