Comparing groups without inflating your error rate, which variant fits which design, and the effect sizes examiners now expect.
T-tests compare two groups; ANOVA compares three or more. The reason ANOVA exists rather than running several t-tests is that every additional comparison raises the chance of finding something by accident, and running six t-tests instead of one ANOVA means your real error rate is far higher than the one you are reporting. Which variant you need follows from how many factors you have, whether the same people appear more than once, and whether you are adjusting for a covariate.
This is the question the whole family answers, and it is worth understanding rather than accepting.
Every significance test carries a chance of a false positive — conventionally set at five per cent. Run one test and that is your risk. Run several on the same data and the chance that at least one comes out significant by accident climbs steeply: with three groups compared pairwise you are running three tests, with five groups you are running ten.
ANOVA handles the comparison in a single test, holding the overall error rate where you set it. It tells you that the groups differ somewhere, but not which ones — which is what post hoc tests are for, and they apply their own corrections for exactly the same reason.
This is also why running many tests across a thesis and reporting the ones that worked is a problem. If you ran twenty comparisons, say so. Reporting three significant results from twenty tests without mentioning the other seventeen is the kind of thing that is difficult to defend once noticed.
| Design | Test | Non-parametric alternative |
|---|---|---|
| Two independent groups | Independent-samples t-test | Mann–Whitney U |
| Same people, two occasions | Paired-samples t-test | Wilcoxon signed-rank |
| One sample against a known value | One-sample t-test | Sign test |
| Three or more independent groups | One-way ANOVA | Kruskal–Wallis |
| Same people, three or more occasions | Repeated-measures ANOVA | Friedman |
| Two or more factors at once | Factorial ANOVA | — |
| Groups, adjusting for a covariate | ANCOVA | — |
| Several related outcomes at once | MANOVA | — |
| Two categorical variables | Chi-square test of independence | Fisher’s exact, for small expected counts |
The commonest design error is using an independent-samples test on repeated measures. If the same participants appear in both conditions, the observations are not independent, and treating them as though they were both wastes the statistical advantage of the design and violates the test’s assumptions.
With two or more factors, you get main effects for each and, more importantly, the interaction: whether the effect of one factor depends on the level of another. The interaction is usually the interesting result, and it changes how the main effects should be read.
Where a significant interaction is present, interpreting main effects on their own can be actively misleading — if a treatment helps one group and harms another, the main effect may be near zero while the treatment matters a great deal. Plot the interaction, and interpret it before the main effects.
Repeated-measures designs are efficient because each participant acts as their own control, which removes between-person variation. They carry an extra assumption, sphericity: that the variances of the differences between all pairs of conditions are roughly equal.
Software typically reports Mauchly’s test for this, though that test is itself sensitive to sample size — underpowered in small samples and liable to flag trivial departures in large ones. Because corrections such as Greenhouse–Geisser cost little when the assumption holds, many methodologists now recommend applying one routinely rather than deciding on the basis of Mauchly’s result. Whichever you do, report it.
Also plan for attrition: repeated-measures analyses typically drop any participant missing a single occasion, so a design with four time points can lose a substantial part of the sample.
ANCOVA compares groups while statistically adjusting for a covariate — commonly a baseline score. Used properly it increases precision.
It carries a requirement that is frequently violated: the covariate must not itself be affected by the independent variable. Adjusting for something that lies on the causal path between the treatment and the outcome removes part of the effect you are trying to measure, and produces a result that cannot be interpreted as intended. In practice this means covariates should be measured before the intervention, or be genuinely unaffected by it.
It also assumes homogeneity of regression slopes — that the relationship between covariate and outcome is similar in each group. Test it, because if slopes differ, the single adjustment ANCOVA makes does not describe any group well.
MANOVA tests several related dependent variables simultaneously. Its legitimate use is when the outcomes are conceptually a set and you are interested in the pattern across them.
Its common misuse is as a way to avoid correcting for multiple tests: running MANOVA and then, on a significant multivariate result, examining each outcome separately without further adjustment. If the outcomes are not genuinely a conceptual set, analysing them separately with an appropriate correction is more honest and easier to defend.
Tests whether two categorical variables are associated. Its main condition concerns expected cell counts, which should not be too small — the conventional guidance is that expected counts below five in more than a small proportion of cells make the test unreliable, and Fisher’s exact test is the usual alternative in small tables.
Chi-square tells you there is an association somewhere in the table, not where or in what direction. Standardised residuals show which cells are driving it, and an effect size such as Cramér’s V or the phi coefficient for a two-by-two table should accompany it.
A significant ANOVA says the groups are not all the same. It does not say which differ, and reporting an ANOVA without following it up leaves the actual question unanswered.
Post hoc tests compare pairs after the fact, with a correction for the number of comparisons. Tukey’s HSD is the common choice for all pairwise comparisons with equal variances; Games–Howell is used where variances differ; Bonferroni is simple, general, and conservative — it controls the error rate at the cost of power, which matters if your sample is modest.
Planned comparisons are stronger where your hypotheses specified particular contrasts in advance. They are more powerful than post hoc tests because you are making fewer comparisons, and specifying them before you see the data is exactly the discipline examiners want to see.
Whichever you use, name the correction and say why. “Post hoc tests were conducted” without specifying which is an invitation to be asked.
Most journals and examiners now expect an effect size alongside every test, because significance alone conflates the size of an effect with the size of your sample.
Interpret them against benchmarks from your own field where they exist, rather than applying generic small, medium and large labels, which were offered as a fallback for situations with no better guidance.
A complete report of a comparison includes: the test used and why, whether its assumptions were checked and what was found, the test statistic with degrees of freedom, the exact p-value, an effect size, group means and standard deviations or medians and interquartile ranges, and the sample size the analysis actually ran on after missing data.
Whenever you are comparing three or more groups. Each separate test carries its own chance of a false positive, so running several inflates your real error rate well above the level you are reporting — with five groups, pairwise comparison means ten tests. ANOVA handles the comparison in one test and holds the overall error rate where you set it, then post hoc tests identify which groups differ, with their own correction applied.
ANCOVA is ANOVA with a covariate statistically controlled, usually to increase precision by removing variation from a known source such as a baseline score. It carries a requirement that catches people out: the covariate must not itself be affected by the independent variable. Adjusting for something on the causal path between treatment and outcome removes part of the effect you are trying to measure, so covariates should be measured before the intervention or be genuinely unaffected by it.
It applies to repeated-measures ANOVA and means the variances of the differences between all pairs of conditions are roughly equal. Software reports Mauchly’s test for it, but that test is sensitive to sample size — underpowered in small samples and prone to flagging trivial departures in large ones. Since corrections such as Greenhouse–Geisser cost little when the assumption holds, many methodologists now suggest applying one routinely. Whichever approach you take, report it.
Tukey’s HSD for all pairwise comparisons when variances are roughly equal; Games–Howell when they are not; Bonferroni when you want something simple and general, accepting that it is conservative and costs power. If your hypotheses specified particular comparisons in advance, planned contrasts are stronger still, because fewer comparisons means more power. Always name the test and correction you used rather than writing that post hoc tests were conducted.
Yes — most journals and examiners now expect them as standard, and a test reported without one looks dated. A p-value confounds the size of an effect with the size of your sample, so a trivial difference reaches significance in a large study while an important one may not in a small study. Use Cohen’s d for t-tests, eta squared or omega squared for ANOVA, and Cramér’s V for chi-square, and interpret them against your own field’s benchmarks where they exist.
Often, yes. The assumption concerns the residuals rather than the raw variable, and with reasonably large and balanced groups these tests tolerate departures from normality fairly well. Where the violation is severe, use the non-parametric equivalent — Kruskal–Wallis for independent groups, Friedman for repeated measures — or a robust alternative, and report what you did and why. Running the original test without mentioning the violation is the one option that is not defensible.
Independent tests used on repeated measures, or covariates that should not have been adjusted for, are quick to spot and awkward to explain afterwards. Describe your design and a PhD in your field will confirm the analysis fits.
Leave an email or a WhatsApp number — whichever you prefer — and tick how we should reach you. We reach out within 30 minutes.
