Statistical Assumptions

Why testing assumptions with a test misleads

The instinct is to run a significance test for each assumption and take the p-value as a verdict. That is less reliable than it looks, for a reason worth understanding.

Tests such as Shapiro–Wilk for normality or Levene’s for equal variances are themselves affected by sample size. In a small sample they lack the power to detect a violation that genuinely matters, so they return a reassuring non-significant result. In a large sample they detect trivial departures that have no practical consequence, flagging a problem where none exists. So they are least informative precisely where you most want guidance.

The practical answer is to look at plots and treat significance tests as supporting information rather than the decision. A histogram and a Q–Q plot tell you more about normality than a p-value; a residuals-versus-fitted plot tells you more about variance and linearity than any test.

It is worth noting too that assumptions are not binary. The question is not whether a condition holds exactly — it never does — but whether the departure is large enough to affect your conclusions. Many procedures tolerate moderate violations reasonably well, and being able to say which ones your analysis is sensitive to is a much stronger position than reporting a table of assumption tests.

Normality — of what, exactly

The most common misconception in doctoral statistics is that regression and ANOVA require your variables to be normally distributed. They do not.

For regression, the assumption concerns the residuals — the differences between observed and predicted values. Your predictors can be any shape at all; skewed predictors are not a violation of anything. Check residuals with a histogram and a Q–Q plot, where points falling close to the diagonal indicate approximate normality.

For t-tests and ANOVA, the assumption strictly concerns the sampling distribution of the mean. With reasonably large samples this tends towards normality regardless of the shape of the underlying variable, which is why these tests are fairly robust in practice. Small samples are where non-normality matters most, and unfortunately where the tests for it are least able to detect it.

Where the violation is genuine and severe, the options are a non-parametric equivalent, a transformation reported explicitly, or a robust method such as a bootstrapped test. Any is defensible. Running the original analysis without mentioning the problem is not.

Equal variances and linearity

Homoscedasticity means the spread of residuals stays roughly constant across the range of predicted values. Where it fails, the coefficients themselves are typically still usable but the standard errors are wrong, which makes your p-values and confidence intervals unreliable.

Diagnose it by plotting residuals against fitted values. Random scatter is what you want; a fan or funnel shape widening across the range is the classic signal. For group comparisons, Levene’s test is the conventional check, with the sample-size caveat above — and adjusted versions of t-tests and ANOVA that do not assume equal variances are available and should simply be reported as such when used. In regression, robust standard errors handle it directly.

Linearity means the relationship you are modelling is actually a straight line. Correlation and linear regression describe straight-line relationships only, so a strong curved relationship can return a coefficient near zero and lead you to report no relationship where a clear one exists. Plot the data before trusting any coefficient; where the relationship is curved, a transformation or a model that allows curvature is the answer.

Independence — the one you cannot patch

Most tests assume observations are independent: one participant’s value tells you nothing about another’s. Unlike the others, this cannot be fixed after the fact by transformation or a robust standard error, because it is a property of the design.

It is violated whenever data are clustered — pupils within classes, patients within hospitals, employees within teams. People inside a cluster resemble each other, so you have less independent information than your row count implies. Analysing clustered data as though it were independent understates the standard errors and overstates significance, sometimes badly.

It is also violated by repeated measurements of the same person, which is why repeated-measures designs need their own tests, and by time series data where consecutive observations are related — the Durbin–Watson statistic is the usual check there.

Where your design produced clustering, the analysis has to account for it: multilevel or mixed-effects models, or cluster-robust standard errors. This is a design-stage decision, and discovering it at analysis is expensive. If you sampled through organisations, schools or wards, you have clustered data whether or not it was planned.

Multicollinearity

Multicollinearity is high correlation among your predictors. It does not bias the model’s overall predictions, but it makes individual coefficients unstable and hard to interpret — standard errors inflate, coefficients can flip sign, and a predictor that matters can appear non-significant.

The tell-tale pattern is a model that is highly significant overall while few individual predictors are, which usually means the predictors are competing to explain the same variance.

The variance inflation factor is the standard diagnostic. Commonly cited thresholds vary considerably between sources, so cite whichever you apply rather than presenting it as a standard. A correlation matrix of predictors is a useful first look, though it will miss collinearity involving three or more variables together, which is what VIF catches.

Remedies, in order of preference: drop one of a pair of near-redundant predictors on theoretical grounds; combine them into a single composite or latent construct if they are measuring much the same thing; centre variables where the collinearity comes from an interaction term, which is a normal and expected consequence rather than a problem; or accept it, if the affected variables are controls whose individual coefficients you were never going to interpret.

Outliers and influential cases

An outlier is an unusual value. An influential case is one whose presence materially changes your results, and they are not the same thing — an extreme value in the middle of the predictor range may barely move the model, while a moderate value at the edge of the range can move it a great deal.

Three diagnostics work together. Standardised residuals identify cases the model fits poorly. Leverage identifies cases with unusual predictor values. Cook’s distance combines both, indicating how much the model would change if a case were removed. Cut-offs for all three are conventions and differ by source, so state and cite the one you use.

What to do about them follows an order. First establish whether the value is an error — data entry, instrument fault, a participant who misread the scale. Errors are corrected or removed and the removal reported. If the value is genuine, removing it because it is inconvenient is not defensible, and it may be among the most informative cases you have. Where an influential genuine case materially changes a conclusion, the honest approach is to report the analysis with and without it and let the reader see the difference.

Decide the policy before looking at what each option does to your results. Deciding afterwards is how analysis drifts into selecting the answer you preferred.

Missing data

Rarely listed as an assumption and frequently the most consequential problem in the chapter, because the default handling in most software quietly changes who your sample represents.

The standard framework distinguishes three situations. Data may be missing completely at random, unrelated to anything — the harmless case, and the rarest. It may be missing at random, where the missingness is explained by other variables you measured. Or it may be missing not at random, where it depends on the unobserved value itself — people with the most severe symptoms declining to report them, for instance. The last is the most damaging and cannot be tested for directly, only argued about.

Listwise deletion, the usual software default, removes any case missing on any variable in the analysis. With several variables each having a little missing data, this can discard a surprising proportion of the sample, and it is only safe when data are missing completely at random — otherwise the remaining sample is systematically different from the one you recruited.

Multiple imputation and full information maximum likelihood are the better-supported approaches under missing-at-random conditions and are available in standard software. Mean substitution, still occasionally taught, artificially reduces variance and is difficult to defend.

Whatever you do, report how much data was missing, where it fell, whether those with missing data differed from those without on variables you did measure, what method you used, and the sample size each analysis actually ran on.