What Is P-Hacking

P-hacking is running several analyses on the same data until one of them comes out statistically significant, usually p < .05, and then reporting that one as if it were the only analysis you ran. The data are real and nothing is faked. The significant result is picked out from among many tries, and the reader is never told how many tries there were.

You will meet it wherever a p-value decides something. An A/B test gets declared a winner, a study gets published, or a dashboard segment gets flagged as the place where a campaign "really worked." In each case the question is whether the result was predicted in advance or found after looking through the data. P-hacking is what happens when it was found and then presented as predicted.

A threshold of p < .05 promises that, when there is no real effect, a single test will cross the line by chance only about 5% of the time. That promise covers one test. Each additional test is another 5% chance of a false alarm, and the chances add up.

The extra tests rarely look like cheating. Researchers call them "researcher degrees of freedom": the choices about which outcome to measure, which subgroup to look at, which outliers to drop, whether to add a control variable, and when to stop collecting data. Any one of these choices can be defended. The trouble comes from making them after seeing the results, and keeping the version that worked.

That mechanism was the subject of Joseph Simmons, Leif Nelson and Uri Simonsohn's 2011 paper "False-Positive Psychology." The same three researchers later coined the term p-hacking in a 2014 paper and run the blog Data Colada, which has audited published work for this problem ever since.

Simonsohn separates two styles. "Slow" p-hacking nudges one analysis step by step, for example dropping observations one at a time, and it leaves a visible pile of p-values just under .05. "Fast" p-hacking jumps between unrelated analyses, such as switching to a different outcome, and leaves no such pile. His simulations show that both damage the findings equally.

There is a limit to how far it goes. In Data Colada's simulations, pushing a null effect from p < .05 down to p < .01 took roughly nine times as many additional analyses. With a finite pool of choices (96 in their scenario), that attempt often fails. This is why a p-hacked result tends to land just under the line rather than far below it.

A study with 40 outcomes. Data Colada's most recent p-hacking post examines a Quarterly Journal of Economics field experiment, "A Cognitive View of Policing," in which officers were trained to "consider different ways of interpreting situations." The paper reported drops in use of force and discretionary arrests. The study tracked about 40 outcome variables. Simonsohn does not accuse the authors of fraud. His point is that with 40 outcomes available, some will look significant by chance, so the headline results need evidence that they were chosen before the data came in.

Commercial A/B tests. A 2018 analysis of 2,101 commercial A/B tests by Berman, Pekelis and colleagues found that about 57% of experimenters p-hacked once results reached 90% confidence, typically by stopping a test early after peeking. According to Convert's summary, this raised the false discovery rate at that threshold from 33% to 42%, meaning more than four in ten declared wins could be noise.

The evidence here is contested. A later study by Alex Miller and Kartik Hosanagar looked at 2,270 experiments from 242 firms on a different e-commerce platform. Their method could detect p-hacking in as few as about 3% of experiments, yet they found essentially none at the platform's 95% threshold. How common p-hacking is seems to depend on the platform and the team. Neither study settles how common it is in A/B testing as a whole.

AI that runs its own analyses. At the 2026 World Conference on Research Integrity, Nihar Shah warned that autonomous "AI scientist" systems can show "automated p-hacking behavior." A June 2026 preprint he co-authored argues that language models can try thousands of analytic variations far faster than a person, and proposes preregistration as the fix.

These problems get grouped together, but they fail in different ways. A fabricated result is a lie about the data. The procrastination study that stayed famous for 24 years is a case of a bad result outlasting its flaws. P-hacking is harder to see because every number in the paper is true.

The file drawer works at the level of whole studies, while p-hacking works inside one. Simonsohn's position is blunt: "P-hacking is easy to stop. File-drawering [is] nearly impossible."

A p-hacked p-value looks exactly like an honest one, which is why the p-value itself cannot tell you whether it was earned. The useful questions are about what happened before the analysis.

Was the metric chosen in advance? Preregistration means writing down the hypothesis, primary metric and analysis before collecting data. Simonsohn says it takes about five minutes on a site like AsPredicted. For an A/B test, the equivalent is fixing one primary metric and the sample size before launch, and treating every other metric as a guardrail.

How many things were tested? Count the outcomes, segments and variants. If several variants were compared, a correction such as Bonferroni should have been applied.

Does it survive variations? Simonsohn lists three defenses that actually rebut a p-hacking charge: proof the choices were made before seeing the data, a result that holds across reasonable alternative analyses, and a direct replication on new data. Explaining why the chosen analysis was reasonable is not enough, because every option a p-hacker picks from is reasonable. If none of the three can be shown, he concludes, the critic is probably right.

One more warning sign is a p-value sitting just under the line, such as .049, on a surprising result from a small sample. That is not proof of anything. It is a reason to ask the three questions above.