Validity & Reliability

Reliability: consistency, and its several forms

Reliability asks whether a measurement is repeatable. It comes in forms that answer different questions, and reporting the wrong one for your design is a small error that signals a larger confusion.

• Internal consistency — do the items in a scale hang together? The most commonly reported, usually as a coefficient across items. Note that this says nothing about whether the scale measures one thing; a scale capturing two constructs can still look consistent.
• Test-retest — does the same person score similarly on two occasions? Relevant for stable traits, meaningless for states that genuinely change. The interval matters and should be justified: too short and people remember their answers, too long and real change is mistaken for unreliability.
• Inter-rater — do two observers or coders agree? Essential wherever judgement is involved in producing the data, and reported with a chance-corrected statistic rather than raw percentage agreement.
• Parallel forms — do two versions of an instrument produce equivalent results? Relevant where alternate versions exist.

The point worth holding onto: reliability is necessary and never sufficient. A bathroom scale that reads six kilos heavy every time is perfectly reliable. Consistency tells you nothing about accuracy.

The four kinds of validity

“Is it valid?” is four questions, and confusing them is why validity sections often ramble.

They trade off against each other, which is worth saying explicitly in a limitations section. A tightly controlled laboratory experiment maximises internal validity and often has poor external validity. A large naturalistic survey has better external validity and cannot establish causation. Recognising the trade-off you made, and why, reads as command of your design.

Threats to internal validity

These are the alternative explanations for your finding, and being able to name the ones relevant to your design is what makes a limitations section convincing rather than apologetic.

• History — something happened between measurements, unrelated to your study, that explains the change.
• Maturation — participants changed simply through time passing: growing older, more experienced, more tired.
• Selection — the groups differed before anything was done to them. The main threat in any non-randomised comparison.
• Attrition — who dropped out was not random. If the people who struggled left, your remaining sample looks better than the intervention made it.
• Testing — taking a measure changed people, through practice or through prompting them to think about the topic.
• Instrumentation — the measure itself changed: a revised questionnaire, an observer who became more skilled, a coder who drifted.
• Regression to the mean — participants selected for extreme scores will tend to score nearer average next time regardless of any intervention. A genuinely common false positive in studies selecting the worst-performing cases.
• Diffusion — the control group learned about or received elements of the intervention.

Randomisation addresses many of these at once, which is why experiments carry the causal claims they do. Where randomisation was impossible, the argument has to be made threat by threat: which apply here, and what makes each implausible in your case.

External validity, and the claim you can make

External validity concerns whether findings extend beyond your study — to other people, settings, times or measures.

The main determinant is sampling. Findings from a convenience sample of students in one institution describe those students; extending to a wider population requires either probability sampling or a theoretical argument about why the mechanism should generalise. The second is legitimate and needs stating as an argument rather than assumed.

Other limits worth naming:

• Setting. Effects found under controlled conditions may not survive real environments, where people are distracted and constrained.
• Time. Findings tied to a particular moment — a policy period, an economic condition, a public health situation — may not hold later, and saying so is honest rather than weak.
• Measures. A relationship established with one instrument may not appear with another measuring the same construct differently.

The strongest move is the same one that works elsewhere: state the boundary of the claim explicitly, and say what would be needed to extend it. A modest claim argued carefully survives examination far better than an ambitious one the design never supported.

Writing the section

Validity sections fail in a predictable way: they define the terms, assert that validity was ensured, and move on. Definitions belong in a textbook, and assertion is not evidence.

A section that works does four things:

1. Names the relevant threats for your specific design. Not all of them — the ones that actually apply.
2. Says what you did about each. Randomisation, matching, a control group, a validated instrument, blinding, pre-testing.
3. Reports the evidence. Reliability coefficients for your own sample rather than quoted from the original validation study, and validity evidence for your measures.
4. States what remains unaddressed, and what that means for the claims.

Two frequent errors worth avoiding. Quoting the reliability from the paper that developed an instrument is not evidence about your data — reliability is a property of a dataset, not of a questionnaire. And using validity and reliability language for qualitative work is a mismatch: that tradition has its own criteria, and applying the wrong vocabulary signals confusion about both.