Linear Regression Assumptions: How to Check Residuals, Homoscedasticity, and Normality

Linear regression assumes five things about the data: (1) LINEARITY — the relationship between predictors and response is linear, not curved; (2) INDEPENDENCE — observations are independent of each other; (3) HOMOSCEDASTICITY — the variance of errors is constant across all predictor values; (4) NORMALITY — the errors (residuals) are approximately normally distributed; (5) NO MULTICOLLINEARITY — predictors in multiple regression are not too highly correlated with each other. Checking these assumptions is the diagnostic phase of regression analysis. The tools are: - Residuals vs fitted plot: checks linearity and homoscedasticity - Q-Q plot of residuals: checks normality - Scale-location plot: checks homoscedasticity more rigorously - Residuals vs leverage (influence plot): detects influential outliers - Variance Inflation Factor (VIF): detects multicollinearity - Durbin-Watson: checks independence of residuals (time series) If assumptions are violated, regression coefficients and p-values may be biased, inefficient, or misleading. The correct response is to identify which assumption is violated and apply the appropriate fix: transform variables, use robust standard errors, use generalized least squares, or switch to a different model (logistic regression for binary outcomes, quantile regression for skewed data). This content is for educational purposes only and supports statistics student learning.

The response variable must have a linear relationship with each predictor. For a curved relationship, linear regression produces biased coefficient estimates and misses the true pattern. How to check: plot residuals against fitted values. If the relationship is linear, residuals should scatter randomly around zero with no pattern. A curve (like a U shape, inverted U, or systematic trend) indicates non-linearity. Worked example: suppose you fit a linear regression and the residuals vs fitted plot shows residuals positive for low and high fitted values, negative for middle fitted values. This is a U-shape — your actual relationship is probably quadratic, not linear. Responses to non-linearity: 1. Transform the response: try log(y) or sqrt(y) if y is positively skewed. These compress the high end and can linearize exponential or power relationships. 2. Transform predictors: try log(x) or x² for relationships that curve. Polynomial regression (adding x² as a predictor) models quadratic relationships directly. 3. Use splines or GAMs: for complex non-linear patterns, generalized additive models (GAMs) fit smooth curves instead of straight lines. 4. Switch to a non-linear model: exponential growth, logistic saturation, power functions — each has its own appropriate model. Important: transforming variables changes the interpretation of coefficients. A coefficient of 0.5 in a log(y) ~ x model means 'a one-unit increase in x is associated with a 50% increase in y' (because log × 100 ≈ percentage). Report on the transformed scale clearly.

Observations must be independent of each other. Independence is violated when measurements are correlated in time (time series) or clustered in space (nested data, repeated measures). Common violations: - Time series data: today's value correlated with yesterday's - Panel data: same subject measured multiple times - Nested data: students within classrooms, patients within clinics - Spatial data: measurements close in space are correlated How to check: Durbin-Watson test for time series detects residual autocorrelation. Durbin-Watson statistic of 2.0 indicates no autocorrelation; below 1.5 suggests positive autocorrelation; above 2.5 suggests negative. More modern tests: Ljung-Box, Breusch-Godfrey. For non-time-series clustering, examine residuals grouped by cluster. If residuals within clusters are systematically positive or negative, cluster correlation exists. Responses to dependence: 1. Generalized Least Squares (GLS): accounts for known correlation structure. 2. Random effects / mixed models: allow intercepts or slopes to vary by cluster. 3. Autoregressive models (AR, ARMA): for time series data. 4. Spatial regression: for spatially correlated data. 5. Cluster-robust standard errors: adjusts standard errors for cluster structure. Why it matters: with correlated residuals, standard errors are typically too small, making p-values artificially significant. Reported significance may be spurious — type I error rate is higher than nominal α.

The variance of errors must be constant across all levels of predictors. Heteroscedasticity (non-constant variance) produces inefficient coefficient estimates and biased standard errors. How to check: residuals vs fitted plot. Homoscedasticity looks like a roughly even spread of residuals across the x-axis. Heteroscedasticity looks like a fan or funnel shape — variance increases (or decreases) with fitted values. More rigorous: scale-location plot (sqrt of absolute residuals vs fitted). A flat, horizontal reference line indicates constant variance. Formal tests: - Breusch-Pagan test: regresses squared residuals on predictors; significant result suggests heteroscedasticity - White test: more general version of Breusch-Pagan - Goldfeld-Quandt: splits data into low and high predictor groups, compares variances Worked example: suppose you're predicting house prices from square footage. Lower-priced houses show small residual variance (the model fits well). Higher-priced houses show larger residual variance (more variability). The residuals vs fitted plot shows a funnel shape. Breusch-Pagan p < 0.05 confirms heteroscedasticity. Responses: 1. Log transformation of response: log(price) often stabilizes variance for price data. 2. Weighted Least Squares (WLS): gives less weight to observations with larger variance. 3. Robust standard errors (Huber-White, HC0-HC3): don't change coefficients but give accurate standard errors. 4. Generalized Linear Models (GLM): use appropriate error distribution (gamma, Poisson, etc.). Consequences: homoscedasticity violations don't bias coefficients but make standard errors wrong. P-values are misleading. Confidence intervals are incorrect. Predictions for specific data points may be unreliable.

Residuals should be approximately normally distributed. This assumption affects the validity of p-values and confidence intervals, especially for small samples. How to check: Q-Q plot (quantile-quantile). Plot residual quantiles against theoretical normal quantiles. Normal residuals produce points along a straight diagonal line. Deviations from the line indicate non-normality — curves suggest skewness, S-shapes suggest heavy or light tails. Formal tests: - Shapiro-Wilk: tests for normality; p > 0.05 suggests normality is plausible - Kolmogorov-Smirnov: compares residual distribution to normal - Anderson-Darling: similar to K-S but more sensitive to tails - Jarque-Bera: tests skewness and kurtosis Important: with large samples, normality tests become overpowered — they reject normality even for trivial departures. With small samples, they lack power — they accept normality even for meaningful departures. Visual inspection via Q-Q plot is often more useful than formal tests, especially for n > 100. Worked example: Q-Q plot shows residuals follow the line for middle quantiles but deviate below for lower quantiles and above for upper quantiles. This is a heavy-tailed distribution — extreme values are more common than normal would predict. Responses: 1. Transform the response variable: log, sqrt, Box-Cox, Yeo-Johnson. These reduce skewness. 2. Use robust regression: minimizes effect of outliers (e.g., M-estimators, MM-estimators). 3. Large samples rely on central limit theorem: for n > 100, coefficient distribution is approximately normal regardless of residual distribution. Mild non-normality is often not a problem. 4. Consider different models: GLM with appropriate error distribution, quantile regression for skewed data. Practical guidance: for samples larger than 100-200, normality violations are usually not catastrophic due to CLT. For small samples (n < 30), non-normality can meaningfully affect inference — take it seriously.

In multiple regression, predictors should not be too highly correlated with each other. High correlation (multicollinearity) makes coefficient estimates unstable and standard errors inflated. How to check: Variance Inflation Factor (VIF). Calculate VIF for each predictor: VIF_j = 1 / (1 - R²_j), where R²_j is the R-squared from regressing predictor j on all OTHER predictors. Interpretation: - VIF = 1: no multicollinearity - VIF = 1-5: acceptable multicollinearity - VIF = 5-10: borderline; consider addressing - VIF > 10: severe multicollinearity; must address Alternative: correlation matrix of predictors. High pairwise correlations (> 0.8 or 0.9) suggest multicollinearity, though VIF captures more than pairwise relationships. Common causes: - Including both x and x² without centering - Including multiple measures of the same concept (e.g., both height in inches and height in cm) - Including an interaction term and its main effects without centering - Subset relationships (e.g., total budget and subset budgets) Responses to multicollinearity: 1. Remove redundant predictors. If two are almost perfectly correlated, drop one. 2. Combine predictors via PCA or factor analysis. 3. Center predictors before computing polynomials or interactions (subtract mean). 4. Ridge regression or LASSO: regularization methods that handle multicollinearity. 5. Accept it if predictions (not coefficients) are the goal: multicollinearity doesn't affect predictive accuracy, only coefficient interpretation. Consequences: coefficients become unstable — small data changes can flip signs. Standard errors are inflated, making individual coefficients appear non-significant even when the overall model is significant. Interpretation becomes misleading — 'controlling for' other predictors becomes meaningless when predictors are nearly redundant.

Standard diagnostic workflow for a linear regression: 1. Fit the model and extract residuals. 2. Look at the four standard diagnostic plots: - Residuals vs Fitted: check linearity and homoscedasticity - Q-Q Plot: check normality - Scale-Location: check homoscedasticity more rigorously - Residuals vs Leverage: identify influential outliers 3. Compute VIF for all predictors. Address any VIF > 5. 4. If residuals show patterns, identify which assumption is violated and apply the appropriate fix. 5. Re-fit after fixes and recheck diagnostics. 6. Report violations, fixes, and final diagnostic results in your analysis writeup. Modern software: - R: plot(lm_model) produces the four standard diagnostic plots automatically. car package provides VIF. lmtest provides formal tests. - Python (statsmodels): model.summary() gives basic diagnostics; scipy.stats.anderson for normality; statsmodels.stats.diagnostic for tests. - Python (scikit-learn): less automatic — need to construct diagnostic plots manually. - SAS/SPSS: diagnostic options available through regression menus. Important caveat: statistical significance of assumption tests is NOT the same as magnitude of violation. For large samples, trivial departures from normality or homoscedasticity can be 'significant' but not practically meaningful. For small samples, meaningful departures may not reach significance. Always combine formal tests with visual diagnostics and judgment.