Type I vs Type II Errors: Worked Examples and Tradeoffs

In any hypothesis test, two errors are possible. Type I error: rejecting the null hypothesis when it is actually true. False positive. Type II error: failing to reject the null hypothesis when the alternative is actually true. False negative. The two errors are NOT symmetric. Type I error rate (alpha) is set by the researcher in advance — typically alpha = 0.05 means 5% chance of falsely rejecting H0. Type II error rate (beta) depends on the actual effect size, sample size, and chosen alpha — it is computed, not set directly. Table of outcomes: | Reality | Decision: Reject H0 | Decision: Fail to Reject H0 | |---|---|---| | H0 True | Type I Error (alpha) | Correct (1 − alpha) | | H0 False | Correct (power = 1 − beta) | Type II Error (beta) | The four cells exhaust all possibilities. Two are correct decisions (lower-right and upper-right). Two are errors. The challenge is balancing the cost of each error type given the context. Which is "worse" depends entirely on context. In medical screening for a treatable cancer, missing the disease (Type II) is far worse than a false alarm (Type I, which can be resolved with confirmatory testing). In a courtroom, convicting an innocent person (Type I if H0 = innocent) is considered worse than letting a guilty person go (Type II). The researcher must understand the cost asymmetry to choose alpha and design sample sizes appropriately.

Alpha and beta are inversely related when sample size and effect size are fixed. Lower alpha (stricter rejection criterion) raises beta (harder to detect a real effect). Higher alpha (laxer rejection criterion) lowers beta (easier to detect a real effect). Visually, this is the overlap between the null distribution (centered at H0) and the alternative distribution (centered at the true effect). The critical value separates "reject" from "fail to reject." Moving the critical value to the right (more conservative) reduces Type I error but increases Type II error. Moving it left does the opposite. The only way to reduce BOTH error types simultaneously is to increase sample size (which sharpens both distributions) or increase the true effect size (which moves the alternative distribution further from the null). These are the only two levers. Statistical power analyses use this principle: given a target effect size and desired alpha and power, compute the required sample size. Common defaults. Alpha = 0.05 is standard in most fields. Power = 0.80 is a common minimum target — meaning 80% probability of detecting a real effect of the specified size. Achieving alpha = 0.05 and power = 0.80 for a moderate effect size typically requires sample sizes in the dozens to hundreds depending on the test.

Example 1: Medical Screening. A new blood test for a treatable cancer. H0: patient does not have cancer. H1: patient has cancer. Type I error: false positive — patient is told they may have cancer, anxiety follows, confirmatory testing reveals the truth. Cost: anxiety + cost of confirmatory test. Type II error: false negative — patient is told they are clear, but they have undetected cancer. Cost: delayed treatment, potential progression of disease. Type II is far more costly. Screening tests are designed with high sensitivity (low beta) at the cost of moderate specificity (higher alpha). The follow-up confirmatory test reduces the cost of false positives. Example 2: A/B Testing. New checkout page vs control. H0: new page has no impact on conversion. H1: new page changes conversion. Type I error: false positive — launching a worse page based on noisy data. Cost: revenue loss, time to detect. Type II error: false negative — failing to detect a real improvement. Cost: missed revenue uplift, slower experimentation cadence. Many teams treat these as roughly symmetric (alpha = 0.05, power = 0.80). High-traffic sites can afford larger sample sizes that reduce both error rates. Example 3: Manufacturing QA. Quality test for a batch of components. H0: batch meets specifications. H1: batch fails specifications. Type I error: false positive — good batch rejected, scrapped or reworked unnecessarily. Cost: production loss + rework. Type II error: false negative — bad batch accepted and shipped to customers. Cost: warranty claims, brand damage, potential safety issues. Type II is usually much more costly in safety-critical industries (aerospace, medical devices). QA processes use tight rejection criteria (low alpha for accepting bad batches = high alpha for falsely rejecting good batches). Example 4: Criminal Trials. H0: defendant is innocent. H1: defendant is guilty. Type I error: false positive — convicting an innocent person. Type II error: false negative — acquitting a guilty person. The "beyond reasonable doubt" standard sets a very low alpha (perhaps 0.01 conceptually). This raises beta — some guilty people are acquitted. The justice system explicitly accepts this asymmetry because the cost of Type I error (wrongful imprisonment) is considered much higher than Type II (guilty person freed). This is the most explicit institutional encoding of the alpha-beta tradeoff.

Power is the probability of correctly rejecting a false null hypothesis. Power = 1 − beta. A test with 80% power has 80% chance of detecting a real effect of the specified size at the specified alpha level. Four drivers of power: (1) Effect size — larger true effects are easier to detect. (2) Sample size — larger samples sharpen the alternative distribution, improving power. (3) Alpha — higher alpha (more liberal rejection criterion) increases power but raises Type I error rate. (4) Variability — lower population variance increases power; higher variance reduces it. Sample size calculation. For a two-sample t-test detecting a medium effect (d = 0.5) with alpha = 0.05 and power = 0.80, the required sample size per group is approximately 64. For a small effect (d = 0.2), the required sample size is approximately 393 per group. For a large effect (d = 0.8), only 26 per group is needed. These rules of thumb come from standard power analysis tables; specific software (G*Power, R, Python statsmodels) computes exact values for various tests. Underpowered studies are a major source of false positives in published research. A study with 30% power that detects an effect has only a 30% chance of finding the true effect — but if it does find an effect, the published result is likely an inflated estimate (because only the larger-than-true effects clear significance). This is the "winners curse" in research. Pre-registration and power calculation are standard mitigations.

Snap a photo of any hypothesis test result or study design and StatsIQ computes the Type I and Type II error rates, identifies the alpha-beta tradeoff for the chosen test, and computes statistical power. For study design, StatsIQ runs power analyses given target alpha and power, computing the required sample size for any common test. For interpreting published results, StatsIQ identifies when reported effect sizes are likely inflated due to low power (the "winners curse" pattern). This content is for educational purposes only.