What Is Statistical Significance? Definition, Examples, and Limits (2026)
Statistical significance means a result is unlikely to have arisen by chance alone, given the assumption that there is no real effect. It is usually declared when the p-value falls below 0.05. That threshold is a convention adopted for convenience, not a property of nature. A significant result is also not the same as an important one.
Those two sentences cover most of what goes wrong with the concept in practice. Teams treat 0.05 as a pass mark and treat significance as proof of impact, and both readings lead to confident decisions built on very little.
This guide covers what the p-value actually answers, where the threshold came from, and how a test is constructed. It ends with the four things significance genuinely cannot tell you.
What is statistical significance?
Statistical significance is a statement about surprise, not about truth. You begin by assuming nothing is happening — that the two groups are identical, or the variables are unrelated. That assumption is the null hypothesis.
Then you ask: if the null hypothesis were true, how often would I see data at least as extreme as mine? If the answer is "rarely," the observed data is hard to reconcile with the assumption, and you reject it.
That is the whole logic. Note what it does not include. There is no statement about how large the effect is, how likely your hypothesis is to be true, or whether the result matters to anyone.
What the p-value actually answers
The p-value is the probability of observing data at least as extreme as yours, assuming the null hypothesis is true.
That conditional clause carries all the weight, and it is almost always dropped. A p-value of 0.03 does not mean there is a 3% chance the result is a fluke. Nor does it mean there is a 97% chance your hypothesis is correct. It means: if there were genuinely no effect, data this extreme would turn up about 3% of the time.
The distinction sounds academic until it changes a decision. Run twenty independent tests on a dataset with no real effects anywhere and you will still get roughly one "significant" result. Five percent of nothing is still something. That is not a flaw in the maths. It is what a 5% threshold means.
Where the 0.05 threshold came from
There is no derivation. The 0.05 cut-off entered common use through convention in early twentieth-century statistical practice and stayed because it was convenient and everyone else was using it.
This matters for how you read borderline results. A p-value of 0.049 and one of 0.051 describe almost identical evidence, yet one gets called significant and the other does not. Treating that boundary as a hard line between real and unreal is the single most common misuse of the concept.
Report the actual p-value rather than only whether it cleared the bar. Readers can then judge for themselves how strong the evidence is.
How significance is tested
The null hypothesis
State the assumption of no effect precisely before looking at the results. "Version B has the same conversion rate as version A" is testable. "Version B is better" is not, because it does not specify what would count as evidence against it.
The test statistic
Choose a test that matches the data. Comparing two group means usually calls for a t-test; comparing proportions calls for a z-test or a chi-square test on a contingency table. Using the wrong test produces a number that looks legitimate and is not.
The p-value and the decision
The test converts your data into a p-value. Compare it against the threshold you set before running the test. Choosing the threshold afterwards, or running the test repeatedly until it clears, invalidates the entire procedure.
Statistical significance vs. practical significance
| Statistical significance | Practical significance | |
|---|---|---|
| Question answered | Could this be chance? | Is this worth acting on? |
| Depends on sample size | Heavily | Not at all |
| Expressed as | p-value | Effect size, confidence interval |
| Can be achieved by | Collecting more data | Only by a real effect |
The last row is the one to remember. With a large enough sample, almost any difference becomes statistically significant, including differences far too small to matter. A conversion improvement of 0.02% across ten million users will clear any threshold you like, and it is still not worth the engineering time.
This is why effect size belongs next to every p-value. Significance says the effect is probably not zero; effect size says whether anyone should care.
What statistical significance does not tell you
How large the effect is. A p-value shrinks with sample size regardless of the size of the underlying difference. It is not a measure of magnitude.
How likely your hypothesis is. The p-value is calculated assuming the null hypothesis holds. It cannot then turn around and tell you the probability that it does not. Answering that question requires a different framework entirely.
Whether the result will replicate. A single significant result in a single sample is weak evidence. Replication is what turns it into a finding.
Whether the study design was sound. Significance testing assumes the data was collected properly. A biased sample produces a confidently significant, confidently wrong answer, and no amount of statistical rigour downstream will repair it.
Common mistakes when reporting significance
Reporting only whether it cleared 0.05. Publish the actual p-value. "p = 0.048" and "p = 0.052" tell a reader far more than "significant" and "not significant." That pair of labels implies a difference in evidence that does not exist.
Running the test until it passes. Checking results daily and stopping when the p-value dips below the threshold guarantees a significant result eventually, whether or not there is an effect. Fix the sample size in advance.
Testing many variants without adjusting. Comparing five variants against a control is five tests, and the chance of at least one false positive is much higher than 5%. Corrections exist for exactly this.
Reading a non-significant result as no effect. An underpowered test finds nothing whether or not something is there. Report the confidence interval so readers can see what effect sizes remain plausible.
Omitting the effect size. Significance says the effect is probably not zero. Only the effect size says whether it is worth doing anything about, and only a confidence interval shows how precisely you have pinned it down.
How to test significance on your own data with Powerdrill Bloom
Step 1: Upload your data
Upload the results file — experiment exports, survey responses or transaction records. Powerdrill Bloom profiles the columns, so group sizes and missing values are visible before any test runs.
Step 2: Describe the test in natural language
State what you are comparing and what kind of measure it is. Compare conversion rates between these two groups, and say whether the difference is significant. Ask for the effect size and the confidence interval alongside the p-value. Ask too whether the test's assumptions hold for this data, rather than assuming they do.
Step 3: Export the chart, report, or deck
Take out the comparison chart, a table with the p-value and confidence interval, or a written summary for the review.
Conclusion
Statistical significance answers one narrow question: could this result plausibly have come from chance alone? It is genuinely useful for that, and useless for everything else people ask of it. The 0.05 threshold is a convention, a significant result can be trivially small, and a non-significant result is not proof that nothing happened.
Read alongside effect size and a confidence interval, it does real work. If you want that trio without setting up the tests by hand, try Powerdrill Bloom on your results file. See also our auto insights page, the guide to analyzing A/B test results without a statistician, running descriptive statistics, and conducting a t-test with AI.
Frequently asked questions
What does statistically significant mean in simple terms?
It means the result would be unlikely to appear if there were no real effect. It does not mean the effect is large or important — only that chance alone is a poor explanation for what you observed.
What is a p-value?
The probability of seeing data at least as extreme as yours, assuming the null hypothesis is true. It is not the probability that your hypothesis is correct, and it is not the chance that the result is a fluke.
Why is 0.05 used as the significance level?
It is a convention from early twentieth-century statistical practice rather than a derived value. Because it is arbitrary, a p-value of 0.049 and one of 0.051 represent nearly identical evidence despite falling on opposite sides of it.
Can a result be significant but not important?
Yes, and it happens constantly with large samples. A big enough sample makes tiny differences statistically significant, which is why effect size should always be reported next to the p-value.
What does it mean if my result is not significant?
That your data does not provide strong evidence against the null hypothesis — not that the null hypothesis is true. An underpowered test can miss a genuine effect entirely, so a non-result is often a statement about sample size.