Reliability vs validity, and why "the validity of the test" is the wrong phrase

Reliability is consistency. A measure is reliable when it gives similar scores each time it is used in the same way: on different days, across its own items, or with different raters.

Validity is about meaning. The 2014 Standards for Educational and Psychological Testing define it as "the degree to which evidence and theory support the interpretations of test scores for proposed uses of tests" (p. 11).

A measure can be reliable without being valid. The reverse is harder, because scores that mostly reflect random error have limited power to predict or to support decisions.

Textbooks and revision guides usually teach validity as a list of types, such as face, content, criterion and construct validity. The Standards, which three American associations prepare jointly, treat validity as one concept supported by several sources of evidence, and they call the unqualified phrase "the validity of the test" incorrect. This article gives the exam answer and the current definition side by side.

• Reliability is the consistency of scores across time, items or raters. Validity is whether the evidence supports the interpretation of the scores for a stated use.
• A ruler held to people's index fingers would give very consistent scores and say nothing about self-esteem. That is the open textbook's example of reliability without validity.
• A correlation of +.80 or more is one textbook's rule of thumb for good reliability. The Standards ask for more precision as the stakes of a decision rise.
• The Standards describe validity as a unitary concept with five sources of evidence, not as separate types.
• Exam specifications still list the types. AQA's A-level specification, for example, names face, concurrent, ecological and temporal validity.

Reliability and validity in one table

Neither is settled by one study. The open textbook Research Methods in Psychology says both are "not established by any single study but by the pattern of results across multiple studies". The quotes here are from its 2nd Canadian edition (2015); later editions exist.

What reliability means

The Standards note that the word has two senses. One is the reliability coefficient of classical test theory, the correlation between scores on two equivalent forms of a test. The other is "the consistency of scores across replications of a testing procedure". The Standards use the term reliability/precision for the general idea (p. 33).

The open textbook splits the general idea into three kinds of consistency, each with its own check.

Reliability applies to more than questionnaires. In the research on defence mechanisms, rating scales applied to interview transcripts reach usable reliability, as do self-report instruments such as the Defense Style Questionnaire. Our article on defence mechanisms sets out the evidence.

The +.80 rule of thumb

For test-retest reliability, the textbook says that "a test-retest correlation of +.80 or greater is considered to indicate good reliability". It gives the same figure for a split-half correlation and for Cronbach's alpha.

Treat +.80 as one textbook's guideline. The Standards link the precision that is needed to the stakes: "the need for precision increases as the consequences of decisions and interpretations grow in importance".

A decision that is not easily reversed, such as rejecting a candidate for professional school, warrants "a higher degree of reliability/precision" (p. 33).

When low reliability is not a problem

Reliability is judged against what the construct is supposed to do. The textbook's example is mood: "The very nature of mood, for example, is that it changes", so a mood measure with a low test-retest correlation over a month "would not be a cause for concern".

The Standards say the same in general terms. If an interpretation assumes that a construct does not vary across occasions, variability across occasions is a potential source of measurement error (p. 33).

Validity as students are taught it

Validity is usually presented as a list of types, and the list depends on the book.

The textbook defines its labels like this.

• Face validity: whether a measure appears "on its face" to measure the construct. The textbook calls it "at best a very weak kind of evidence", partly because it rests on people's intuitions about behaviour, "which are frequently wrong". In the MMPI-2, items such as "I enjoy detective or mystery stories" and "The sight of blood doesn't frighten me or make me sick" both measure the suppression of aggression, although neither looks related to it.
• Content validity: whether a measure "covers" the construct. If test anxiety is defined as nervous feelings plus negative thoughts, the measure needs items on both. It is checked against the definition of the construct, not by a statistic.
• Criterion validity: whether scores correlate with other variables, called criteria, that they should correlate with. A new test-anxiety measure should correlate negatively with exam performance. It is concurrent validity when the criterion is measured at the same time as the construct, and predictive validity when it is measured later.
• Convergent and discriminant validity: a new measure should correlate with existing measures of the same construct (convergent) and not with measures of conceptually distinct variables (discriminant). A new self-esteem measure, the textbook says, should not correlate very highly with mood. If it did, it might be measuring mood instead.

The textbook adds that "a good way to interpret these types is that they are other kinds of evidence" for judging the validity of a measure. The Standards go further.

Validity as the Standards define it

The Standards are prepared jointly by the American Educational Research Association, the American Psychological Association and the National Council on Measurement in Education. The current edition dates from 2014 and has been open access since March 2021. Its definition of validity is the sentence quoted at the start of this article. A few lines later it states the rule behind this article's title.

No interpretation travels everywhere: "No test permits interpretations that are valid for all purposes or in all situations" (p. 23). The Standards' example is a mathematics achievement test. Evidence that its scores support placing students in courses "does not permit inferring validity for other purposes", such as promotion or teacher evaluation (p. 14).

The Standards also drop the list of types. On page 14 they write: "Validity is a unitary concept. It is the degree to which all the accumulated evidence supports the intended interpretation of test scores for the proposed use."

The edition "refers to types of validity evidence, rather than distinct types of validity", and it does not use the "historical nomenclature (i.e., the use of the terms content validity or predictive validity)". The 2014 text says that the 1999 edition did the same.

The Standards name five sources of validity evidence: test content, response processes, internal structure, relations to other variables, and the consequences of testing. A source is needed only where a proposed interpretation depends on it (p. 14). The table sets the textbook labels beside these sources.

Construct validity fits no single row. In the Standards every proposed interpretation includes specifying "the construct the test is intended to measure" (p. 11), so construct validity corresponds roughly to the whole framework, not to one source of evidence.

This is the position of the joint Standards, and not every textbook has adopted its wording, as the three lists above show. A Joint Committee to revise the Standards was named in June 2024, according to the announcements on testingstandards.net, so a new edition is in preparation. The 2014 text is the one cited here.

Exam specifications still list the types

AQA's A-level specification (version 1.4, for teaching from September 2025) lists, under research methods, "Types of validity across all methods of investigation: face validity, concurrent validity, ecological validity and temporal validity". For a question about the types, use the labels that your own specification lists. The Standards help with the other kind of question, the one that asks whether a measure is valid. The answer to that is always: valid for which interpretation, and on what evidence?

Can a test be reliable but not valid?

Yes. The textbook's example is a ruler held to people's index fingers as a measure of self-esteem. It "would have extremely good test-retest reliability", but "absolutely no validity".

A less absurd case is a bathroom scale that reads 2 kilograms too heavy every time. Its readings are consistent and wrong.

The Standards give the formal version. Systematic errors, such as an incorrect answer key or test forms that have not been adequately equated for difficulty, affect scores "in a consistent rather than a random manner". They are not counted as a lack of reliability/precision. They are "construct-irrelevant factors that reduce validity but not reliability/precision" (p. 36).

Can a test be valid but not reliable?

Not at the extreme. The Standards say that the level of reliability/precision "has implications for validity". To the extent that scores reflect random errors of measurement, "their potential for accurate prediction of criteria, for beneficial examinee diagnosis, and for wise decision making is limited" (pp. 34 and 35). Adequate score reliability is also on the Standards' list of components of validity evidence (p. 22).

Short of the extreme, the two trade off. The Standards note that performance assessments that depend on ratings of extended responses tend to have lower reliability than more structured assessments such as multiple-choice tests, but "can sometimes provide more direct measures of the attribute of interest".

The Standards add that "some of these sacrifices in reliability/precision may reduce construct irrelevance or construct underrepresentation and thereby improve the validity of the intended interpretations of the scores" (p. 36). Consistency is needed as evidence but is not enough on its own, and how much is enough depends on the use.

Validity of a study is a different question

Psychology also uses the word validity for whether a study supports its conclusions. Internal validity and external validity belong to this family. They describe a study, not the scores of a questionnaire. The textbook discusses them in its chapter on experiments, where it lists internal, external, construct and statistical validity as four to consider when judging whether an experiment is sound. The label construct validity appears in both families, which is one reason the terms get mixed.

A third meaning is informal. A "reliable finding" in everyday speech is one that holds up when other researchers repeat the study. That is replication, a separate question from the reliability of a measure. Our article on which well-known effects survived replication covers it for ten cognitive biases.

Drilling the terms against each other

Reliability, validity and their sub-types are near-synonyms that exams test by definition. Our guide to memorising psychology terms argues that terms like these have to be learned against each other, not one at a time. In Psychology World you save terms to a list and drill the list with flashcards, multiple choice or typing. The Question game is multiple choice, which makes it a recognition test, and it needs at least four words on the list. The Typing game asks you to produce the answer yourself. The study modes are described on the home page.