Aiken's V: how to validate your instrument with expert judgment

If your thesis uses your own or an adapted questionnaire, your advisor will probably ask you to validate it with expert judgment before collecting data. The idea is simple: several people who know the topic read each item and rate it. If they agree that the item is clear, relevant and measures what it should, you have evidence of content validity. Aiken's V is the most common way to sum up that agreement in one number.

What judges rate

Every university has its own form, but most use a version of these four criteria, on a 1 to 4 scale:

• Sufficiency: the items of a dimension are enough to measure it. It is rated once per dimension, not per item.
• Clarity: the item is easy to understand and its wording is appropriate.
• Coherence: the item is logically related to the dimension or indicator it measures.
• Relevance: the item is important and should be included.

These criteria come from Escobar-Pérez and Cuervo-Martínez (2008), widely cited in Latin American theses. If your form uses others (pertinence, wording, objectivity), the calculation is the same.

How many judges you need

Theses usually have between 3 and 10 judges, and many universities ask for at least 3 or 5. More judges give a more stable V, but they are also more work to gather. Pick people with experience in the topic or in research (lecturers, practitioners, methodologists) and note their degree, specialty and years of experience, since the form usually asks for them.

How Aiken's V is calculated

The formula is V = S / (n × (c − 1)). S is the sum of each judge’s rating minus the lowest value on the scale, n is the number of judges and c is the number of values on the scale.

An example with 5 judges and a 1 to 4 scale. The clarity ratings of an item are 4, 4, 3, 4 and 3. Subtracting the minimum (1) leaves 3, 3, 2, 3 and 2, so S = 13. The denominator is 5 × (4 − 1) = 15. The clarity V of that item is 13 / 15 = 0.87.

V goes from 0 to 1. A value of 1 means every judge gave the highest rating. The most common cut-off is 0.80, although some universities accept 0.70 and others also ask for the Penfield and Giacobbi (2004) confidence interval, so that the lower bound also passes the cut-off. Check what your guidelines say.

How it works in Thesis Mode

In SurvIQ you load your instrument with its variables, dimensions and items (by hand or by reading your thesis draft with AI) and add your judges. Each judge can rate through a link, without an account, and sign on screen. If your university requires its own signed paper form, you enter the scores by hand or paste them from Excel, and upload the scanned form.

When the judges finish, the matrix builds itself: V per item and criterion, each item’s mean, V per criterion and the overall V. Items below the cut-off are highlighted in yellow, together with the judges’ comments.

What to do with an item that fails

• Read the comments. In the example, two judges said "as if they were objects" was offensive to respondents, which is why clarity and coherence dropped.
• If the problem is wording, rewrite the item. SurvIQ keeps the previous version so the change is on record.
• If the item is unrelated to the dimension, remove it or move it to the right dimension.
• If your university requires it, run a second round with only the revised items.

For your appendices you can download the matrix in Excel and in Word with the method note, a form per judge with their details, scores, comments and signature, and the operationalization matrix.

What SurvIQ does not do

SurvIQ does not write your thesis or decide for you. It calculates V, lays out the tables and shows which items to review. The interpretation, the validity section and the decisions about each item are yours and your advisor’s.

Thesis Mode is available from the Basic plan. If you only need the calculation, SurvIQ's Aiken's V calculator is free and needs no account.