Statistical Power and Sample Size: Beating Type II Error

Power is the probability of correctly rejecting the null hypothesis when a true effect exists. In equation form: power = 1 − β, where β (beta) is the probability of Type II error (failing to reject a false null). Type I error (alpha, α): rejecting a true null. You conclude there is an effect when there isn't one. Convention: α = 0.05 (5% false positive rate). Type II error (beta, β): failing to reject a false null. You miss a real effect. Convention: β = 0.20 (20% chance of missing a true effect), giving power = 0.80 (80% chance of detecting a true effect). Why power matters: a study that lacks power is likely to produce false negatives. It's a waste of time and money if conducted anyway, and its null result is uninterpretable (you can't conclude "no effect" from a non-significant result in an underpowered study). Power analysis is the planning step that prevents this. The four linked quantities: sample size (n), alpha (α), effect size (d, η², f², r, etc.), and power (1 − β) are mathematically linked. Fix three, and the fourth is determined. This is the foundation of sample size planning — specify the effect size you want to detect, the alpha you'll use, and the power you want, and the math tells you how many subjects you need.

Effect size quantifies the magnitude of an effect independent of sample size. A large effect is easy to detect; a small effect requires many more observations. Common effect size measures: Cohen's d (for comparing two means): d = (mean₁ − mean₂) / pooled SD - Small effect: d = 0.2 - Medium effect: d = 0.5 - Large effect: d = 0.8 - Example: if group means differ by 5 points and pooled SD is 10, d = 0.5 (medium). Eta squared (η²) for ANOVA: η² = SS_effect / SS_total, the proportion of variance explained. - Small: 0.01 - Medium: 0.06 - Large: 0.14 - Example: if factor explains 10% of variance, η² = 0.10 (between medium and large). Cohen's f² for multiple regression: f² = R² / (1 − R²), measuring unique variance explained. - Small: 0.02 - Medium: 0.15 - Large: 0.35 Correlation coefficient (r) for correlation tests: - Small: 0.10 - Medium: 0.30 - Large: 0.50 Phi or Cramér's V (for chi-square): - Small: 0.10 - Medium: 0.30 - Large: 0.50 Choosing effect size for planning: 1. From prior similar research: use published effect sizes in your field. 2. Minimum effect of interest: smallest effect that would be practically meaningful to detect. Often more honest than "what do I expect." 3. Cohen's conventions (small/medium/large) as defaults if no other information is available. 4. Pilot data can provide an estimated effect size, but small pilots give unreliable effect size estimates. The critical insight: if you use Cohen's conventions, pick based on what size effect is worth detecting. Don't plan for a large effect just because it makes the sample size small — if the true effect is smaller than your planning assumption, you'll be underpowered.

Research question: does a new teaching method improve exam scores compared to standard lecture? Plan an independent-samples t-test. Inputs: - Expected effect size: d = 0.5 (medium — based on prior literature) - Alpha: 0.05 (two-tailed) - Desired power: 0.80 - Test: two-sample t-test, equal n per group Formula for approximate required n per group: n per group ≈ ((z_α/2 + z_β)² × 2) / d² With α = 0.05 (two-tailed), z_α/2 = 1.96. With power = 0.80, β = 0.20, z_β = 0.84. n per group ≈ ((1.96 + 0.84)² × 2) / 0.5² ≈ (2.80² × 2) / 0.25 ≈ (7.84 × 2) / 0.25 ≈ 15.68 / 0.25 ≈ 62.72 Round up: n = 63 per group, N = 126 total. Verification with software: G*Power with inputs (t-test, two independent samples, d = 0.5, α = 0.05, power = 0.80) gives n = 64 per group. The small discrepancy comes from software using t-distribution critical values rather than normal approximation. What happens if the effect is smaller than planned: - d = 0.3 instead of 0.5: required n per group = ((2.80)² × 2) / 0.09 ≈ 174 per group - d = 0.2 instead of 0.5: required n per group ≈ 393 per group This illustrates why overestimating effect size is dangerous — you end up underpowered. If you plan for d = 0.5 and the truth is d = 0.3, your 63-per-group study has only about 38% power, meaning you're more likely to miss the effect than detect it. Rule of thumb: when in doubt, assume a smaller effect and plan for a larger sample. Or report confidence intervals around your estimated effect, which remain interpretable regardless of whether the result reaches significance.

Research question: does exam score differ across 4 teaching methods? Inputs: - Expected effect size: f = 0.25 (medium) - Alpha: 0.05 - Desired power: 0.80 - Test: one-way ANOVA with 4 groups Cohen's f for ANOVA is the standard deviation of group means divided by within-group SD. It's related to eta-squared by: f² = η² / (1 − η²). Medium f = 0.25 corresponds to η² ≈ 0.06. Using G*Power (one-way ANOVA, f = 0.25, α = 0.05, power = 0.80, 4 groups): Required total N ≈ 180, so ~45 per group. As effect size varies: - f = 0.10 (small): total N ≈ 1200, ~300 per group - f = 0.25 (medium): total N ≈ 180, ~45 per group - f = 0.40 (large): total N ≈ 76, ~19 per group Key observation: the number of groups also matters. More groups = more total N required, even at the same per-group n. A 6-group ANOVA at f = 0.25 needs ~216 total, not just 180 × 6/4. Considerations: - Unbalanced designs: use the harmonic mean of planned group sizes in power calculations. - Unequal variances: adjust for the violation or use Welch's ANOVA (slightly lower effective power). - Post-hoc pairwise comparisons inflate the effective sample size needed if you plan multiple tests — apply Bonferroni or Holm correction.

Research question: is hours studied correlated with exam score? Inputs: - Expected effect size: r = 0.30 (medium) - Alpha: 0.05 (two-tailed) - Desired power: 0.80 - Test: Pearson correlation Fisher's z transformation is used for power calculations on correlations. Required N can be approximated as: N ≈ ((z_α/2 + z_β) / Fisher_z(r))² + 3 Fisher_z(0.30) = 0.5 × ln((1 + 0.30) / (1 − 0.30)) = 0.5 × ln(1.857) = 0.5 × 0.619 = 0.310 N ≈ ((1.96 + 0.84) / 0.310)² + 3 ≈ (2.80 / 0.310)² + 3 ≈ (9.03)² + 3 ≈ 81.6 + 3 ≈ 85 So detecting r = 0.30 with 80% power at α = 0.05 (two-tailed) requires approximately 85 observations. Sensitivity: - r = 0.10: need ~782 observations - r = 0.30: need ~85 observations - r = 0.50: need ~30 observations Correlation is particularly sensitive to sample size in power calculations because the variability of r depends on r itself — smaller correlations have much larger sampling variance relative to their magnitude. Effect size inflation warning: small samples produce noisy estimates of r. An r = 0.40 observed in n = 20 has a 95% CI of approximately (−0.04, 0.72) — the population r could easily be 0 or could be 0.70. Small studies often overestimate effect sizes. Planning future studies based on small-sample effect sizes frequently leads to underpowered replications.

Mistake 1: running a post-hoc power calculation based on the observed effect size. This is statistically meaningless. If you found a non-significant result, post-hoc power calculated from your effect size will always look low — you're essentially computing the p-value again in different units. Use a priori power analysis (planning before data collection) based on expected or minimum-meaningful effect sizes, not post-hoc analysis based on observed effects. Mistake 2: using effect sizes from small pilot studies. Small pilots produce highly variable effect size estimates. An r = 0.50 in a pilot of n = 10 might truly be r = 0.20 or r = 0.70 in the population. Don't scale your main study based on a small pilot's point estimate — either use the lower confidence bound of the pilot, or use theoretical expected effect sizes instead. Mistake 3: assuming a large effect without justification. "I expect a large effect because the intervention is strong" is not justification. Unless you have prior similar research showing large effects, default to medium (or small for novel interventions). Overly optimistic effect size assumptions are the leading cause of underpowered studies. Mistake 4: ignoring attrition. If 20% of participants will drop out, your planned n = 100 becomes effective n = 80. Account for attrition: plan enough participants so that post-attrition n meets power requirements. Mistake 5: multiple comparisons not built into the calculation. If you plan 5 comparisons, each at α = 0.05, your family-wise error rate is much higher than 5%. Apply Bonferroni (divide alpha by number of tests) at the planning stage, which increases required sample size. A Bonferroni-corrected alpha of 0.01 requires ~40% more sample than α = 0.05 for the same power. Mistake 6: treating power analysis as a one-time calculation. As effect sizes, dropout rates, and design details evolve during pilot and early data collection, update your sample size plan. Many studies discover partway through that their design assumptions were wrong and need revision. Mistake 7: running it once in software and not checking. G*Power and similar tools can give very different results based on which specific test you select. Two-sample t-test, one-sample t-test, paired t-test, and Welch's t-test all have different power formulas. Confirm you're using the correct test option.