Why doesn’t my result match SPSS?

SPSS shows the two-tailed p by default ("Exact Sig. (2-tailed)"). For expert judgment what matters is whether agreement is above chance, which is the one-tailed p. The table shows both.

My university averages the p-values. Do you have that?

Some theses average the p-values of all items. It is common practice but has no statistical basis. Here you get the overall test using every answer, which does, plus each item’s value.

What if the judges used a 1–5 scale?

Then use Aiken’s V or Hernández-Nieto’s CVC, which make use of the whole scale.

Do I test each criterion separately?

Yes. If judges rated clarity, relevance and appropriateness, run one test per criterion.

All tools

Binomial test for expert judgment

When your judges answer "yes" or "no" for each item (for example, whether it is clear, relevant or appropriate), the binomial test tells you whether their agreement is higher than chance. Paste the answers from Excel: one row per item and one column per judge, with 1 for "yes" and 0 for "no". You can also paste "Yes" and "No" as they are.

Paste the answers (rows = items, columns = judges; 1 = yes, 0 = no)

The calculation runs in your browser. We don’t store what you type.

How it’s calculated

For each item, count how many judges said yes (x) out of k judges.

If judges answered at random, each "yes" would have a probability of 0.5. The p-value is the probability of getting x or more "yes" answers by chance: P(X ≥ x).

The calculation is exact, with the binomial distribution, not an approximation.

The overall test pools every answer for every item and applies the same calculation.

How to read the result

p < 0.05 on an item: the judges’ agreement is significant and the item is considered valid.
With 5 judges, an item needs all 5 "yes" answers to be significant (p = 0.031). With 4 judges not even full agreement gets there (p = 0.063), which is why at least 5 are recommended.
Review or rewrite items with p ≥ 0.05, especially if a judge left a comment.

Frequently asked questions

Why doesn’t my result match SPSS?: SPSS shows the two-tailed p by default ("Exact Sig. (2-tailed)"). For expert judgment what matters is whether agreement is above chance, which is the one-tailed p. The table shows both.
My university averages the p-values. Do you have that?: Some theses average the p-values of all items. It is common practice but has no statistical basis. Here you get the overall test using every answer, which does, plus each item’s value.
What if the judges used a 1–5 scale?: Then use Aiken’s V or Hernández-Nieto’s CVC, which make use of the whole scale.
Do I test each criterion separately?: Yes. If judges rated clarity, relevance and appropriateness, run one test per criterion.

Know what you want to ask?

Build the survey in SurvIQ, share it by link, WhatsApp or QR code, and watch results come in live. The Free plan is free forever.

Create a free survey

More tools

Cohen’s and Fleiss’ kappa calculatorMeasures agreement between raters or judges, beyond what chance would produce.Normality test (Shapiro-Wilk and Kolmogorov-Smirnov)Check whether your data is normally distributed before choosing a statistical test.Student’s t-test calculatorCompares the means of two groups or a before and after, with Levene’s test and Cohen’s d.Mann-Whitney U test calculatorCompares two independent groups when the data aren’t normal or are ordinal.Wilcoxon signed-rank test calculatorCompares a before and after of the same people when the data aren’t normal.One-way ANOVA calculatorCompares the means of three or more groups, with Levene, Welch, eta squared and pairwise comparisons.