Hypothesis Testing: The Complete Guide With 6 Worked Tests

Hypothesis testing is a statistical decision procedure for using sample data to evaluate two competing claims about a population: a null hypothesis (typically "no effect" or "no difference") and an alternative hypothesis (the research claim). The procedure computes a test statistic from the sample, compares it against a reference distribution under the null, and produces a p-value — the probability of observing a test statistic at least as extreme as the one observed if the null hypothesis were true. If the p-value is below a pre-specified significance level (alpha, typically 0.05), the null is rejected in favor of the alternative; otherwise the null is not rejected. Hypothesis testing does NOT prove the alternative is true. It only quantifies how surprising the observed data would be under the null. Modern practice pairs hypothesis testing with effect-size reporting and confidence intervals because a statistically significant result with negligible effect size is not practically meaningful.

Every hypothesis test, regardless of test family, follows the same seven-step framework. Memorize this sequence and the workflow becomes automatic. 1. State the null and alternative hypotheses (H0 and H1) symbolically (e.g., H0: μ = 100 vs H1: μ ≠ 100). 2. Choose the significance level (alpha) — typically 0.05, sometimes 0.01 or 0.10 depending on the field and the cost of type I errors. 3. Select the appropriate test based on the data type, sample size, and assumption checks (covered in the decision tree below). 4. Compute the test statistic from the sample data using the test's formula. 5. Determine the p-value (or critical value) from the reference distribution. 6. Compare the p-value to alpha and make a reject / fail-to-reject decision. 7. State the conclusion in plain language tied to the original research question. A common mistake is collapsing step 1 into the conclusion — writing the alternative as "the new drug is better" instead of "μ_new > μ_old". Symbolic statements force precision and prevent ambiguous "improvement" claims that the data may not support.

Use a one-sample test when comparing a sample mean to a known or hypothesized population value. Use the z-test when the population standard deviation (σ) is known (rare in practice); use the t-test when σ is unknown and must be estimated from the sample (the common case). Formula: z or t = (x̄ − μ_0) / (s / √n) For t-test, degrees of freedom = n − 1. Worked Example. A factory claims its production process produces parts averaging 100 mm in length. A sample of n = 25 parts has mean x̄ = 102 mm and standard deviation s = 5 mm. Test at alpha = 0.05 whether the process mean differs from 100 mm. 1. H0: μ = 100; H1: μ ≠ 100 (two-tailed). 2. alpha = 0.05. 3. One-sample t-test (σ unknown, sample is roughly normal). 4. t = (102 − 100) / (5 / √25) = 2 / 1 = 2.0. df = 24. 5. From the t-distribution, two-tailed p-value for t = 2.0 with df = 24 is approximately 0.057. 6. p-value (0.057) > alpha (0.05) — fail to reject H0. 7. Conclusion: At alpha = 0.05, we do not have sufficient evidence to conclude the process mean differs from 100 mm. The result is borderline; collecting more data or accepting alpha = 0.10 would change the conclusion. Borderline results like this one demonstrate why effect size and confidence intervals matter. The 95% CI for μ is x̄ ± t_critical × (s/√n) = 102 ± 2.064 × 1 = (99.94, 104.06) — narrowly includes 100, consistent with the fail-to-reject decision but suggesting the true mean is most likely above 100.

Use a two-sample (independent samples) t-test to compare means from two unrelated groups. Variants exist for equal vs unequal variances; modern practice uses Welch's t-test (unequal variances) by default because it is robust to variance violations and reduces to Student's t-test when variances are equal. Formula (Welch): t = (x̄₁ − x̄₂) / √(s₁²/n₁ + s₂²/n₂) Worked Example. Two teaching methods are compared on student test scores. Method A: n₁ = 20, x̄₁ = 78, s₁ = 8. Method B: n₂ = 25, x̄₂ = 73, s₂ = 10. Test at alpha = 0.05 whether the methods differ. 1. H0: μ_A = μ_B; H1: μ_A ≠ μ_B (two-tailed). 2. alpha = 0.05. 3. Welch's two-sample t-test (independent groups, unequal variances assumed). 4. t = (78 − 73) / √(64/20 + 100/25) = 5 / √(3.2 + 4.0) = 5 / √7.2 = 5 / 2.683 = 1.864. Welch df ≈ 42.4. 5. Two-tailed p-value for t = 1.864 with df ≈ 42 is approximately 0.069. 6. p-value (0.069) > alpha (0.05) — fail to reject H0. 7. Conclusion: No significant difference between methods at alpha = 0.05. Effect size (Cohen's d) ≈ 0.55, a moderate effect — suggesting the test was underpowered. A power analysis would indicate that detecting a moderate effect at this alpha requires roughly n = 52 per group. Again, the borderline result raises power and effect-size questions. The methods may genuinely differ but the sample is too small to detect the difference reliably. Reporting the effect size makes this clear; reporting only "p > 0.05" hides it.

Three more high-frequency tests with worked one-pass examples. Paired t-test. Use when the same subjects are measured twice (before/after, matched pairs). Compute the difference for each pair, then test whether the mean difference equals zero. Example: 15 subjects measured before and after intervention; mean difference = 4.2, SD of differences = 6, n = 15. t = 4.2 / (6/√15) = 4.2 / 1.549 = 2.71, df = 14, two-tailed p = 0.017. Reject H0 at alpha = 0.05 — intervention had a significant effect. One-way ANOVA. Use to compare means across 3 or more groups. Test statistic is F = MS_between / MS_within. Example: three teaching methods, total n = 60 (20 per group), MS_between = 50, MS_within = 12.5. F = 50/12.5 = 4.0, df = (2, 57), p ≈ 0.024. Reject H0 — at least one method differs. Follow up with Tukey's HSD to find which pairs differ. Chi-square test of independence. Use for two categorical variables. Test whether they are independent. Example: 200 voters surveyed by party (D/R/I) and policy preference (yes/no). Compute expected counts under independence, then χ² = Σ (O − E)² / E. If χ² = 12.5 with df = 2 (= (rows − 1)(cols − 1)), p ≈ 0.002 — strong evidence of association between party and preference. Each of these has its own assumption checklist (paired t-test: normality of differences; ANOVA: normality + equal variances per group; chi-square: expected counts ≥ 5 per cell). Violations push you to non-parametric alternatives covered next.

A practical decision tree based on the question, data type, and number of groups. Is the response variable continuous? YES. How many groups? • One group, comparing to a known value → one-sample t-test (or z if σ known). • Two groups, independent → Welch's two-sample t-test (or Mann-Whitney U if non-normal). • Two groups, paired → paired t-test (or Wilcoxon signed-rank if non-normal). • Three or more groups, independent → one-way ANOVA (or Kruskal-Wallis if non-normal). • Two factors → two-way ANOVA. • Repeated measures across 3+ time points → repeated-measures ANOVA or mixed model. NO (categorical response). What kind of question? • Test of independence between two categorical variables → chi-square test. • Goodness-of-fit to a theoretical distribution → chi-square goodness-of-fit. • Two proportions → two-proportion z-test. Bivariate continuous, testing relationship → Pearson correlation (parametric) or Spearman/Kendall (non-parametric); regression for prediction. The parametric vs non-parametric branch turns on assumption checks: shapiro-wilk for normality, levene's test for equal variances, sample size considerations (CLT helps when n > 30 per group). Non-parametric tests have lower power but are robust to violations.

A p-value is the probability, ASSUMING the null hypothesis is true, of observing a test statistic at least as extreme as the one observed. It is NOT the probability that the null hypothesis is true. This distinction matters because the most common misinterpretation ("p = 0.04 means there is only a 4% chance the null is true") is wrong. Four outcomes are possible in any hypothesis test: | Reality | Decision: Reject H0 | Decision: Fail to Reject H0 | |---|---|---| | H0 is true | Type I error (alpha) | Correct (1 − alpha) | | H0 is false | Correct (power, 1 − beta) | Type II error (beta) | Type I error (α) is wrongly rejecting a true null — a false positive. Convention sets alpha at 0.05. Type II error (β) is wrongly failing to reject a false null — a false negative. Conventions for beta target 0.20 (power = 0.80). Power (1 − β) is the probability of correctly rejecting a false null. It depends on alpha, effect size, and sample size. Increasing sample size is the only lever that increases power without inflating type I error. A published power lookup table for two-sample t-tests at alpha = 0.05, power = 0.80: | Effect size (Cohen's d) | Required n per group | |---|---:| | 0.20 (small) | 393 | | 0.30 | 175 | | 0.40 | 99 | | 0.50 (medium) | 64 | | 0.60 | 45 | | 0.70 | 33 | | 0.80 (large) | 26 | | 1.00 | 17 | This table is the cure for underpowered studies. Compute the smallest effect size of practical interest, look up the required n, and collect that much data — anything less wastes effort and produces ambiguous results.

Hypothesis testing problems span every introductory statistics course, the AP Statistics exam, and the first month of every applied research program. The decision tree (which test) and the assumption checks (which form) are exactly where most students get stuck. Snap a photo of any hypothesis testing problem and StatsIQ identifies the appropriate test, checks assumptions if data is provided, computes the test statistic and p-value, and produces the final reject / fail-to-reject conclusion in plain language. For power and sample-size calculations, StatsIQ produces the required n given alpha, power, and effect size.

Six errors recur. First, conflating "fail to reject H0" with "accept H0" — failing to reject just means the data does not provide enough evidence against H0, not that H0 is true. Second, p-hacking: running multiple tests and reporting only the significant ones, or adjusting alpha after seeing the data. The fix is pre-registration of analysis plans. Third, ignoring effect size: a statistically significant result with tiny effect size is not practically meaningful, and a non-significant result with large effect size signals an underpowered study. Fourth, using a one-tailed test post hoc to make a non-significant two-tailed test significant. The directional choice must be made before seeing the data and justified by theory. Fifth, applying parametric tests without checking assumptions: if data is severely non-normal and n is small, switch to a non-parametric test. Sixth, reporting "p = 0.000" — p-values are never exactly zero. Report p < 0.001 instead.