Sample Size & Power Analysis

What power actually means

A study can go wrong in two directions. It can find an effect that is not real, or it can miss one that is. The first risk is what your significance level controls. The second is what power addresses, and it is the one doctoral projects routinely ignore until it is too late.

An underpowered study is a genuine problem rather than a technicality. It is likely to return a non-significant result whether or not the effect exists, which means a null finding tells you almost nothing — you cannot distinguish “no effect” from “not enough participants to see it”. Worse, when an underpowered study does reach significance, the effect it reports is usually inflated, because only unusually large sample estimates could clear the threshold.

The four quantities are connected: significance level, power, effect size and sample size. Fix any three and the fourth follows. In practice you set the significance level by convention, choose the power you want, decide the smallest effect worth detecting, and solve for the sample size. Conventional power targets circulate widely and differ between fields, so state and cite the level you are aiming for rather than treating any figure as standard.

The hard part: what effect would matter

Everything in a power calculation is straightforward except this input, and it is the one examiners ask about.

You are not guessing what effect you expect to find. You are deciding the smallest effect that would be worth detecting — the point below which a difference would be too trivial to change anything in your field. That is a substantive judgement about your discipline, not a statistical one.

Reasonable ways to arrive at it, roughly in order of strength:

• Practical significance. What size of difference would actually change a decision, a policy or a practice? This is the strongest justification because it is grounded in what your research is for.
• Meta-analyses in your field. The average effect across studies of similar phenomena, cited.
• Comparable published studies using similar measures in similar populations, with the caveat that published effects tend to be larger than true effects, since larger effects are more likely to be published.
• Field conventions, cited, as a last resort. Generic small, medium and large labels were offered precisely as a fallback for situations with no better information, and using them where better information exists is weak.

One approach to avoid: running a small pilot and using its observed effect size to power the main study. Effect estimates from small pilots are extremely imprecise, and building a sample size calculation on one gives the appearance of rigour without the substance. Pilots are for testing procedures, instruments and timings — which is genuinely valuable, and a different job.

Running the calculation

G*Power is free, widely accepted, and covers most designs a doctoral project will use. R packages and commercial software do the same job; for complex designs such as multilevel models or structural equation models, simulation-based approaches are often the only realistic route.

Whichever tool you use, the analysis has to be tied to the specific test you will run. The sample size for an independent-samples t-test is not the sample size for a factorial ANOVA with an interaction, and an interaction typically needs considerably more participants than a main effect. Running a generic calculation and applying it to a different analysis is a common shortcut that does not survive questioning.

Then add to the number, for reasons that are entirely predictable:

• Attrition. Longitudinal and intervention studies lose participants, and the calculation gives you the number you need to analyse, not to recruit. Estimate the loss from comparable studies and recruit above the target accordingly.
• Missing data. Listwise deletion in a model with several variables can remove more cases than people expect.
• Clustering. If participants are grouped inside schools, wards or teams, they carry less independent information than their count implies, and the required sample rises — sometimes substantially. This needs building into the calculation rather than noticing afterwards.

In the methodology chapter, report the test, the effect size you targeted and where it came from, the significance level, the power, the resulting number, and how attrition was accounted for. The reasoning is what is being assessed, not the figure.

Why post hoc power tells you nothing

This one is worth knowing precisely, because supervisors and reviewers still occasionally request it.

After a non-significant result, it is tempting to compute the power the study had, using the effect size you actually observed. Software will produce a number. That number is not informative, and the reason is straightforward: computed this way, power is a direct mathematical function of the p-value you already have. A non-significant result will always produce low observed power, and a significant result will always produce high observed power. It cannot tell you anything the p-value did not, and it certainly cannot tell you whether your study was well designed.

The methodological literature is clear about this, and the practice has been criticised for decades. If you are asked for it, the constructive response is to offer something that does answer the underlying question: a confidence interval around your observed effect. That shows the range of effects your data are consistent with, and if that interval excludes effects large enough to matter, you have genuine evidence of absence rather than merely absence of evidence. If it includes large effects, your study simply could not resolve the question — which is the honest conclusion.

Power analysis belongs before data collection, where it can change what you do. Afterwards, intervals do the work.

When you cannot recruit the number you need

This is a common position in doctoral research, particularly with clinical populations, senior professionals or small organisations. The calculation says four hundred; access realistically gives you ninety.

Pretending otherwise is the worst option. Better ones, depending on circumstances:

• Change the question to one the achievable sample can answer. A study that estimates a relationship precisely in a narrower group beats an underpowered attempt at a broader claim.
• Simplify the analysis. Fewer predictors, fewer groups and no interaction terms all reduce what you need.
• Use a repeated-measures design where possible. Each participant acting as their own control removes between-person variation and can markedly reduce the sample required.
• Reframe the study as estimation rather than testing. Report effect sizes with confidence intervals and be explicit that the study was not powered for hypothesis testing. This is increasingly accepted and is far more defensible than a significance test the design could never support.
• State it plainly as a limitation, with the power the study did have for the effect you considered meaningful, calculated in advance.

Examiners deal with constrained samples constantly. What they respond badly to is a candidate who did not notice, or who noticed and covered it.