Confidence Interval Explained

Your A/B test dashboard says the new checkout page converts at 4.2%, with a 95% confidence interval of 3.6% to 4.8%. Does that mean there is a 95% chance the real conversion rate is somewhere between 3.6% and 4.8%? Almost everyone reads it that way. Strictly speaking, it does not mean that.

A confidence interval is a range built around a sample estimate to show how much that estimate could move if you had drawn a different sample. The "95%" describes the method, not this particular range. If you ran the same test again and again and built an interval the same way each time, about 95% of those intervals would contain the true conversion rate. Your one interval either contains it or it does not, and you will never know which.

This page is for analysts, marketers, product managers and researchers who see confidence intervals in dashboards, papers and reports and want to know what they can safely say about one. The mistake most people make is the one above: treating the 95% as a probability about the single interval in front of them. It is the misreading statisticians flag first, and it leads to several others.

Every confidence interval has the same shape: a sample estimate, plus or minus a margin of error.

The sample estimate is the number you measured, such as a conversion rate, an average order value or a difference between two groups. The margin of error is how far the interval stretches on each side. It is built from two parts:

• The standard error, which measures how much the estimate would bounce around from sample to sample. For an average, it is the standard deviation of the data divided by the square root of the sample size. For a proportion like a conversion rate, the standard deviation part is the square root of p × (1 − p), where p is the rate.
• A critical value, a multiplier taken from the normal distribution (a z-score) or the t-distribution (a t-score) that sets how wide the net is. For 95% confidence with a z-score, the multiplier is 1.96.

Put together:

Interval = estimate ± (critical value × standard error)

Three things move the width, and the Statsig guide to 95% intervals lists all of them: larger samples make the interval narrower, more variable data makes it wider, and a higher confidence level (99% instead of 95%) makes it wider while a lower one (90%) makes it narrower.

What the 95% refers to

Think of the method as a net you throw at a fish you cannot see. The fish stays put; it is the true value, and it does not move. Each sample gives you a new throw. A 95% method is one that lands around the fish on about 95 throws in 100. Once a particular throw has landed, the fish is either in the net or it is not. There is no chance left about it.

That is the frequentist definition, and the Cross Validated answer on this exact question puts it plainly: the definition "does not permit one to conclude from a single 95% CI that there is a 95% chance that the mean falls somewhere within the CI." The true value is treated as fixed, not random, so no probability attaches to it once the interval is calculated.

GraphPad's Prism guide draws the line in a slightly different place, and the difference is worth knowing. It says it is correct to say "there is a 95% chance that the confidence interval you calculated contains the true population mean," but "not quite correct" to say there is a 95% chance the population mean lies within the interval, because the mean is fixed: "There is no chance about it" (GraphPad). The safest reading of that first sentence is about the interval before the data come in: the procedure has a 95% chance of producing an interval that catches the true value. After the numbers are on the screen, the stricter sources say no probability is left to assign.

Here is the A/B test from the opening, worked through by hand. The numbers are hypothetical but realistic: a variant of a checkout page shown to 4,300 visitors, 181 of whom bought something. Doing it once by hand makes every dashboard interval you see afterward easier to read.

What that interval lets you claim

You can say: the test measured a 4.2% conversion rate, and the data are reasonably compatible with true rates from about 3.6% to 4.8%, using a method that captures the true value in about 95% of repeated experiments.

You cannot say: there is a 95% chance the true rate is between 3.6% and 4.8%. You cannot say 95% of visitors convert at a rate in that range, because individual visitors either buy or do not. And you cannot say a rerun of the test will land inside 3.6% to 4.8% 95% of the time; as the next section shows, it usually will not.

What a product manager usually wants to know is whether the variant beats the control. The interval above answers a different question: how precisely did we measure the variant on its own? For the comparison, you need an interval around the difference between the two arms, which the next section works through.

The standard reference here is a 2016 paper in the European Journal of Epidemiology by Sander Greenland, Stephen Senn, Kenneth Rothman, John Carlin, Charles Poole, Steven Goodman and Douglas Altman, "Statistical tests, P values, confidence intervals, and power: a guide to misinterpretations" (vol. 31, issue 4, pages 337 to 350). It catalogues 25 misinterpretations across p-values, significance tests, confidence intervals and power, and it is open access; an annotated full text is on Fermat's Library. Its starting point is blunt: no interpretation of these tools is at once simple, intuitive, correct and foolproof.

The confidence interval entries in that list, plus two from other sources, are the ones that show up in workplace reports.

1. "There is a 95% chance the true value is in this interval." This is Greenland et al.'s misinterpretation #19, and Statistics Solutions marks the same sentence FALSE in its list of misconceptions. How to recognize it: any sentence that attaches a probability to one already-calculated range. What to do: say the data are compatible with values in the range, and that the method captures the true value 95% of the time. If you need the probability statement, you need a Bayesian credible interval, which is built differently.

This misreading is not confined to beginners. In a 2023 r/statistics thread on the most confidently incorrect statements people had heard in the field, commenters brought it up unprompted, and at least two said they read intervals that way themselves. Another replied with the correction: "once an interval is calculated, this interval either covers the parameter value or it does not."

2. "95% of the data fall inside the interval." GraphPad warns about this one specifically. A confidence interval is about the population average or rate, not individual observations. How to recognize it: someone points at the interval for average order value and says most orders fall in that range. With a large sample, the interval for the average can be narrow while individual orders vary widely. What to do: if you want the spread of individual values, report the standard deviation or a range of the data, and label it as that.

3. "The intervals overlap, so there's no difference." This is misinterpretation #21. Two intervals can overlap and the difference between the groups can still be statistically significant. A direct test, or an interval for the difference itself, is required.

Using the hypothetical numbers from before: the variant converts at 4.2% (95% CI 3.6% to 4.8%) on 4,300 visitors. Suppose the control converts at 5.2% on another 4,300 visitors, giving a 95% CI of about 4.5% to 5.9%. The intervals overlap between 4.5% and 4.8%. But the difference is 1.0 percentage point, its standard error is √(0.0031² + 0.0034²) ≈ 0.0046, and its 95% interval runs from about 0.1 to 1.9 points. That interval excludes zero. Eyeballing overlap would have told you "no difference"; the right calculation says the control is ahead, though the low end of 0.1 points is close enough to zero that the size of the gap is poorly pinned down. The relationship between an interval that excludes zero and a significant p-value is laid out in the page on how p-values and confidence intervals relate.

4. "95% of future studies will land inside this interval." Misinterpretation #22. Your interval is centered on your estimate, which is itself off from the truth by some unknown amount, and a future estimate will be off by its own amount. The chance that a rerun's estimate lands inside your interval is, in the words of Scientifically Sound's summary of the paper, usually "much less than 95%". How to recognize it: someone treats the interval as a forecast for next quarter's test. What to do: expect reruns to scatter more than that, and do not treat a rerun outside the range as proof something broke.

5. "Values outside the interval have been ruled out." Misinterpretation #20. A value just outside the interval is only slightly less compatible with the data than one just inside it, and that compatibility depends on the data and on the model's assumptions being right. What to do: treat the edges as soft. A true lift a hair beyond the upper limit has not been refuted.

6. "It excludes zero, so it's precise." Misinterpretation #23. Precision is about width. An interval of 0.1% to 40% excludes zero and tells you almost nothing about the size of the effect. As the paper puts it: "It is not a matter of inclusion or exclusion of the null or any other value." What to do: read the width first, then where the interval sits.

7. Treating a confidence interval as a credible interval. This is the Bayesian conflation underneath misreading 1. A credible interval does license "95% probability the true value is in here," but it comes from a different calculation that combines the data with a prior belief. Some A/B testing tools report one, some the other. What to do: check your tool's documentation for which it is before you write the sentence in a deck.

Small samples. Below about 30 observations, or when you do not know the population standard deviation, Statsig's guidance is to use a t-score instead of a z-score. The t multiplier is larger than 1.96 for small samples, so the interval comes out wider, which is the honest answer when you have little data. Finding the t-value from a table is the same lookup covered in calculating a p-value by hand.

A different confidence level. 95% is a convention. GraphPad notes that 99% widens the interval and 90% narrows it, as the worked example shows. Some teams run product tests at 90% to move faster. That is a choice, and the report has to say which level was used, because a 90% interval and a 95% interval from the same data are different ranges.

Comparing two groups. The interval for each arm of a test describes that arm. A claim about which arm wins needs an interval for the difference, as in misreading 3. The same goes for before-and-after comparisons on a dashboard.

Results that were picked out of many. If the interval you are reading is the best-looking one from a dozen segments or a dozen metrics, its 95% coverage no longer holds in practice. Greenland et al.'s guideline (b) asks readers to look at hidden assumptions about how results were generated and selected for presentation, and guideline (g) asks reports to describe the full sequence of analyses, including the plan made in advance and everything actually run. If you cannot see that sequence, discount the interval.

Forecasts that do not report intervals at all. Election forecasts are a live example as of October 2026. Nate Silver's Silver Bulletin model forecasts the House, Senate and governors' races by simulation, about 40,000 runs each time, and describes its uncertainty as a "normal-sized polling error" rather than a stated ±X points at 95% (Silver Bulletin). Decision Desk HQ's 2026 forecast release gave point win probabilities and seat projections, such as 227 House seats for Democrats, without a numeric range in the release itself (DDHQ). A win probability is not a confidence interval, and a seat projection with no range attached hides how wide the plausible outcomes are. If you are quoting one of these numbers, check the methodology page for the range before you treat the point as precise.

The main cost of a confidence interval is sample size. The margin of error shrinks with the square root of the sample, so every halving of the interval's width takes four times as much data. For an A/B test, that means four times the traffic or four times the run time. Going from 95% to 99% confidence costs width in the other direction: in the worked example, it stretched the range from 3.6% to 4.8% out to 3.4% to 5.0%, with no new data.

What it buys you is a sense of scale that a bare estimate or a bare p-value cannot give. Greenland et al.'s first guideline is that correct interpretation means looking at effect-size estimates and their confidence limits, not only whether p falls above or below 0.05. Statsig's guidance for experimenters says the same: use intervals alongside p-values and effect sizes, not in isolation. In Statsig's own example, a 5% conversion lift with a 95% interval of 2% to 8% tells you the effect is likely positive and roughly how large it might be. A p-value alone tells you neither size nor range; the page on what a p-value tells you covers what it does say.