P-Value Interpretation: Common Mistakes and Correct Reading

A p-value is the probability of observing data as extreme as (or more extreme than) the observed data, assuming the null hypothesis is true. Symbolically: p = P(Data ≥ observed | H0 true). Three critical parts of this definition. (1) It is a conditional probability — conditional on H0 being true. (2) "As extreme or more extreme" — it includes the observed value and everything further from the null. (3) It is computed for data, not for hypotheses. The p-value is a property of the data given the model, not a property of the hypothesis given the data. This definition has direct consequences for interpretation. A p-value of 0.03 means: "If H0 were true, there would be a 3% chance of seeing data this extreme or more extreme." It does NOT mean "There is a 3% chance that H0 is true." It does NOT mean "There is a 97% chance that H1 is true." Inverting the conditional probability requires Bayes' theorem and prior probabilities — which p-values do not contain. The American Statistical Association in 2016 published a position statement clarifying these points after decades of misuse. The takeaway: p-values measure compatibility of data with the null hypothesis, not the truth of the null hypothesis.

Mistake 1: "p = 0.03 means there is a 3% chance the null is true." Wrong. The p-value is conditional on H0 being true. Inverting requires Bayes' theorem. The probability that H0 is true depends on prior probability — which the p-value does not include. Mistake 2: "p = 0.03 means there is a 97% chance the alternative is true." Wrong, for the same reason. The complement of the p-value is not the probability of the alternative. Both probabilities require a prior. Mistake 3: "p < 0.05 proves the effect is real." Wrong. A small p-value provides evidence against H0 but does not prove the alternative. Replication, effect size, and study design all matter. A statistically significant result with tiny effect size may have no practical importance. Mistake 4: "p > 0.05 proves there is no effect." Wrong. Failing to reject H0 is not the same as accepting H0. The data may simply be underpowered to detect a real effect. "Absence of evidence is not evidence of absence" applies directly here. Bonus mistake 5: comparing p-values across studies. p = 0.04 and p = 0.06 are essentially the same. The 0.05 threshold is arbitrary; treating studies as different categorically based on p crossing 0.05 misrepresents continuous evidence. Recent recommendations move toward reporting exact p-values, effect sizes, and confidence intervals jointly.

P(H0 true | data) = P(data | H0) × P(H0) / P(data). This is Bayes' theorem. The p-value provides P(data | H0). To get P(H0 | data), you need P(H0) — the prior probability that H0 is true before seeing the data. Without the prior, the inversion cannot be done. A p-value of 0.05 paired with a high prior (say P(H0) = 0.99) yields a high posterior probability that H0 is still true. The same p-value paired with a low prior yields a low posterior. The same data can support opposite conclusions depending on the prior. This is why frequentist hypothesis testing does not produce statements about the probability that H0 is true. It produces statements about the data given the hypothesis. Bayesian analysis explicitly incorporates priors and produces direct posterior probability statements — but this is a different framework with different inputs and assumptions.

A 95% confidence interval and p < 0.05 are mathematically related. For a two-sided test, the test will reject H0 at alpha = 0.05 if and only if the 95% confidence interval does not include the null value (typically zero or some null-hypothesized mean). Worked example. Suppose we test H0: mean = 0 with sample data. If the 95% CI is [0.4, 2.3], the interval does not include zero — the test rejects at p < 0.05. If the 95% CI is [-0.2, 1.8], the interval includes zero — the test fails to reject. The CI and p-value give equivalent information. Why confidence intervals are often preferred for reporting. (1) They directly show effect size and precision. (2) They make non-significant results interpretable — a CI of [-0.05, 0.10] tells the reader the effect is small, while p = 0.30 alone reveals nothing about effect size. (3) They are less prone to the "significance vs effect size" confusion. The recommendation from contemporary statistics: report the confidence interval along with (or instead of) the p-value.

Snap a photo of any test result and StatsIQ produces the correct interpretation, the confidence interval, the effect size, and flags common misinterpretation patterns. For exam prep, the app produces practice problems with multiple-choice interpretation questions and identifies which interpretations are correct versus which are common misunderstandings. StatsIQ also handles Bayes theorem inversions for students who want to understand posterior probability under specified priors. This content is for educational purposes only.