Cohen’s and Fleiss’ kappa calculator
Kappa measures how much two or more raters agree when they classify the same things (for example, judges deciding whether an item is relevant: yes or no, or two observers coding behaviors), after removing the agreement chance alone would produce. Paste one row per case and one column per rater. Categories can be words or numbers. With 2 columns you get Cohen’s kappa; with 3 or more, Fleiss’ kappa.
With 2 columns, Cohen’s kappa is calculated; with 3 or more, Fleiss’ kappa.
The calculation runs in your browser. We don’t store what you type.
How it’s calculated
κ = (Po − Pe) / (1 − Pe), where Po is the observed agreement and Pe the agreement expected by chance from how often each rater uses each category.
Cohen (1960) is for exactly 2 raters. The standard error and confidence interval follow Fleiss, Cohen and Everitt (1969), as in SPSS.
Fleiss (1971) extends the idea to 3 or more raters who rate every case. Kappa per category is also shown, so you can see where disagreement is highest.
The p-value tests whether agreement is better than chance (null hypothesis κ = 0).
How to read the result
- Landis and Koch (1977): below 0 poor, 0 to 0.20 slight, 0.21 to 0.40 fair, 0.41 to 0.60 moderate, 0.61 to 0.80 substantial and 0.81 to 1 almost perfect.
- For expert validation, a kappa of 0.61 or more is usually expected.
- High observed agreement with a low kappa happens when almost everyone picks the same category: chance explains much of the agreement.
Frequently asked questions
- Cohen or Fleiss?
- Cohen only works for 2 raters. With 3 or more judges, use Fleiss. The calculator picks automatically from the number of columns you paste.
- Does it match SPSS?
- Yes. Cohen’s kappa, its asymptotic standard error and p-value match Analyze > Descriptive Statistics > Crosstabs > Kappa. Fleiss’ kappa matches SPSS 26 and later.
- Can I use ordered scales (1 to 5)?
- Yes, but this kappa treats each value as a separate category: a 4 against a 5 counts as much disagreement as a 1 against a 5. For ratings on a scale, Aiken’s V or the CVC are usually a better fit.
Know what you want to ask?
Build the survey in SurvIQ, share it by link, WhatsApp or QR code, and watch results come in live. The Free plan is free forever.
Create a free survey