How to choose a test from your question and your data, what a p-value does and does not tell you, and why effect size matters more than significance.
Choosing a statistical test is not a lookup exercise: it follows from what you are asking, how many groups or variables are involved, what type of data you have, and whether observations are independent. Once those four things are settled, the choice is usually narrow. The harder work is checking that the conditions the test depends on actually hold, and then interpreting the result without claiming more than it supports — which is where most doctoral statistics goes wrong.
Statistics does two different jobs, and confusing them causes trouble later.
Descriptive statistics summarise the data you actually have: means, medians, standard deviations, frequencies, correlations. They make no claim beyond your sample and require no assumptions to be true.
Inferential statistics use your sample to say something about a wider population. That extra step is what brings in assumptions, probability and uncertainty — and it only makes sense if your sample was obtained in a way that supports inference to that population. This is worth pausing on: most doctoral samples are convenience or purposive, and inferential tests run on them still produce output, which quietly encourages a population claim the sampling never earned. Running the test is fine; describing the population it generalises to honestly is the part that needs care.
Nearly every results chapter should begin descriptively. Looking at distributions, spread and missing data before testing anything catches most of the problems that would otherwise be discovered by an examiner.
Four questions settle it in most cases:
That gives you something like this. It covers the common cases rather than every possibility, and the right-hand column matters as much as the left.
| Question | Data and structure | Usual test | If assumptions fail |
|---|---|---|---|
| Do two independent groups differ? | Continuous outcome | Independent-samples t-test | Mann–Whitney U |
| Did the same people change? | Continuous, two occasions | Paired-samples t-test | Wilcoxon signed-rank |
| Do three or more groups differ? | Continuous outcome | One-way ANOVA | Kruskal–Wallis |
| Same people, three or more occasions? | Continuous, repeated | Repeated-measures ANOVA | Friedman |
| Two categorical variables related? | Counts in categories | Chi-square test of independence | Fisher’s exact, for small counts |
| Are two continuous variables related? | Continuous, roughly linear | Pearson correlation | Spearman’s rho |
| Does a set of variables predict an outcome? | Continuous outcome | Multiple linear regression | Robust or transformed models |
| Does a set of variables predict a yes/no outcome? | Binary outcome | Logistic regression | — |
| Do items group into the constructs I expect? | Multiple scale items | Factor analysis — exploratory or confirmatory | — |
| Does a network of relationships fit? | Several constructs at once | Structural equation modelling | — |
Two frequent trip-hazards. Data collected from the same person on several occasions, or from people nested inside groups such as classes or hospitals, breaks the independence assumption — and needs a method that accounts for the nesting rather than a standard test. And Likert-type items are ordinal individually; whether summed scale scores may be treated as continuous is a genuine, live disagreement in the literature, so the defensible move is to state your position and cite it rather than to assume either side has settled it.
Every inferential test depends on conditions. Software will run happily when they are violated and give you a number that looks exactly like a valid one, which is precisely why this section exists.
When an assumption fails you have three honest options: use a test that does not require it, transform the data and say so, or report the violation and discuss what it means for your conclusions. What is not defensible is silence.
More doctoral vivas turn on this than on any other statistical point, and the common misreadings are worth stating precisely.
A p-value is the probability of obtaining a result at least as extreme as the one you observed, if the null hypothesis were true. That is all it is.
It is not:
The conventional 0.05 threshold is a convention, not a law of nature, and treating it as a bright line produces two errors. A result at 0.049 is not meaningfully different in evidential terms from one at 0.051, and reporting the first as a discovery and the second as nothing is not defensible. Meanwhile “approaching significance” is a phrase examiners enjoy asking about.
Significance also depends heavily on sample size. With a large enough sample, a difference far too small to matter will reach significance. With a small sample, an important effect may not. This is why significance alone is never an adequate report of a result.
An effect size says how big the difference or relationship is, in a form that does not depend on your sample size. A confidence interval says how precisely you have estimated it. Together they answer the question a p-value cannot: does this matter, and how sure are we?
Most journals and most examiners now expect them as standard, and reporting a test without them looks dated.
Interpreting effect sizes carries one caution worth stating. The widely quoted labels — small, medium, large — were offered as rough conventions for situations where no better benchmark existed, and their author was explicit that they were a last resort. What counts as a large effect differs enormously between fields: a small effect on mortality can matter far more than a large effect on a laboratory task. Wherever your field has its own benchmarks, use those and cite them, rather than applying generic labels as though they were universal.
A confidence interval also deserves care in wording. A 95% interval is a statement about the procedure — that intervals constructed this way capture the true value 95% of the time in the long run — rather than a 95% probability that this particular interval contains it. Examiners in quantitative disciplines do ask.
A reader should be able to reconstruct what you did without asking you. In practice that means reporting, for each analysis: what was tested, the test used and why, whether its assumptions were checked and what was found, the test statistic with degrees of freedom, the exact p-value rather than a threshold, an effect size, and an interval where applicable. Then the sample size the analysis actually ran on, which after missing data is often not the number you recruited.
Two further habits protect you. State which analyses were planned in advance and which were exploratory — both are legitimate, but presenting the second as the first is not. And where you ran many tests, say how many, because the more comparisons made, the more likely a chance result appears among them; whether and how to adjust for that is a judgement you should make explicitly rather than leave unmentioned.
Your discipline’s reporting standard — whichever style or checklist your field uses — specifies the details. Following it exactly is the cheapest credibility available.
Work through four questions: are you looking for a difference, a relationship or a prediction; how many groups or variables are involved; what type of data do you have; and are the observations independent or repeated. Those four almost always narrow it to one or two candidates. If they do not, the research question is usually still too vague to imply a test, and sharpening it will settle the matter faster than searching for the test will.
It is the probability of getting a result at least as extreme as yours if the null hypothesis were true. It is not the probability that the null hypothesis is true, not the probability your hypothesis is correct, and not a measure of how large or important the effect is. Because it depends heavily on sample size, it should always be reported alongside an effect size and, where appropriate, a confidence interval.
First check what the test actually assumes — for many common procedures it concerns the residuals or the sampling distribution rather than the raw variable, and with larger samples those tests tolerate departures reasonably well. If the assumption genuinely fails, you can use a non-parametric equivalent, transform the variable and report that you did, or use a robust method. Any of those is defensible; running the original test and not mentioning it is not.
A single item is ordinal. Whether a summed or averaged multi-item scale can be treated as continuous is genuinely debated, with respectable arguments and published work on both sides. The defensible approach in a thesis is to state your position, cite support for it, and where the decision materially affects your conclusions, show that the result holds under both treatments. That is a far stronger answer in a viva than assuming the question is settled.
Significance tells you a result is unlikely under the null hypothesis; it says nothing about whether the effect is big enough to care about. In a large sample, a difference too small to be of any consequence will often be significant. Effect sizes and confidence intervals are what let you speak to importance, and interpreting them against benchmarks from your own field — rather than generic small, medium and large labels — is what makes the interpretation credible.
Rules of thumb circulate widely and disagree with each other, which is a good reason not to rely on one. The defensible approach is a power analysis based on the smallest effect that would matter in your field, the number of predictors in your model, and the power you want. Tools such as G*Power do the arithmetic; the reasoning behind the inputs is what your methodology chapter has to show.
Assumption problems and mismatched tests are straightforward to fix before write-up and awkward afterwards. Describe your design, your variables and what you are testing, and a PhD in your field will go through it with you.
Leave an email or a WhatsApp number — whichever you prefer — and tick how we should reach you. We reach out within 30 minutes.
