Inter-Rater Agreement Calculator
Calculate Cohen's Kappa & Percent Agreement
2×2 Contingency Table (Frequencies)
Enter the number of items categorized by Rater A and Rater B.
| Rater A | Rater B | |
|---|---|---|
| Yes / Positive | No / Negative | |
| Yes / Positive | ||
| No / Negative | ||
Calculation Results
Total Observations (N):
0
Observed Agreement (Po):
0%
Expected Agreement (Pe):
0
Cohen's Kappa (κ):
0.000
Interpretation will appear here.
How to Calculate Inter-Rater Agreement
Inter-rater agreement (or inter-rater reliability) is a critical statistical measure used to assess the degree to which different raters/observers give consistent estimates of the same phenomenon. While simple percent agreement tells you how often raters agreed, it fails to account for agreement occurring simply by random chance.
Why Cohen's Kappa?
This calculator uses Cohen's Kappa (κ), a robust statistic that measures inter-rater agreement for qualitative (categorical) items. It is generally considered a more robust measure than simple percent agreement calculation, as $\kappa$ takes into account the possibility of the agreement occurring by chance.
Understanding the Inputs
To use this calculator, you need to populate a 2×2 confusion matrix (contingency table) based on your data:
- Cell A (Agreed Yes): Both Rater A and Rater B said "Yes" or "Positive".
- Cell B (Disagreement): Rater A said "Yes", but Rater B said "No".
- Cell C (Disagreement): Rater A said "No", but Rater B said "Yes".
- Cell D (Agreed No): Both Rater A and Rater B said "No" or "Negative".
Interpretation of Results
Cohen's Kappa ranges from -1 to +1, where +1 indicates perfect agreement. The standard interpretation (Landis & Koch, 1977) is:
- ≤ 0: No agreement (or negative agreement)
- 0.01 – 0.20: Slight agreement
- 0.21 – 0.40: Fair agreement
- 0.41 – 0.60: Moderate agreement
- 0.61 – 0.80: Substantial agreement
- 0.81 – 1.00: Almost perfect agreement
The Formulas
The calculation involves two main steps:
1. Observed Agreement ($P_o$):
$P_o = (a + d) / N$
2. Expected Agreement ($P_e$):
This is calculated based on the marginal totals of the matrix to determine the probability of random agreement.
3. Kappa ($\kappa$):
$\kappa = (P_o – P_e) / (1 – P_e)$