Calculate agreement between two raters for binary outcomes
Enter the counts of observations for Rater 1 vs Rater 2 in the Confusion Matrix below:
Rater 2
Yes / Positive
No / Negative
Rater 1
Total Observations (N):0
Observed Agreement (Po):0%
Expected Agreement (Pe):0%
Cohen's Kappa (κ):0.000
How to Calculate Inter-Rater Reliability in SPSS
Inter-rater reliability (IRR) is a crucial statistical measure used to assess the degree of agreement between different judges, coders, or measuring instruments. When subjective data is collected—such as doctors diagnosing a patient or researchers coding interview data—it is vital to prove that the classification is consistent.
The most common metric for calculating IRR for categorical items (e.g., Yes/No, Positive/Negative) is Cohen's Kappa. This article explains how to calculate it manually using the tool above and provides a step-by-step guide for performing the analysis in IBM SPSS Statistics.
Understanding Cohen's Kappa
Simple percent agreement can be misleading because two raters might agree purely by chance. Cohen's Kappa ($\kappa$) corrects for this chance agreement.
$\kappa = 1$: Perfect agreement.
$\kappa = 0$: Agreement is exactly what you would expect by chance.
$\kappa < 0$: Agreement is worse than chance (systematic disagreement).
Step-by-Step: How to Calculate Cohen's Kappa in SPSS
While the calculator above gives you a quick result for a 2×2 matrix, SPSS allows you to handle larger datasets with multiple cases effortlessly. Follow these steps:
1. Prepare Your Data
Ensure your data is structured correctly in the SPSS Data View:
Rows: Each row represents a specific subject or case (e.g., Patient ID).
Columns: You need two columns representing the raters.
Variable 1: Rater_A_Score (e.g., 0 for No, 1 for Yes)
Variable 2: Rater_B_Score (e.g., 0 for No, 1 for Yes)
2. Navigate the Menu
Go to the top menu bar and select:
Analyze > Descriptive Statistics > Crosstabs…
3. Set Up the Crosstabs
In the Crosstabs dialog box:
Drag Rater_A_Score into the Row(s) box.
Drag Rater_B_Score into the Column(s) box.
It does not matter which rater goes into rows or columns; the Kappa value will be the same.
4. Select the Kappa Statistic
Click the Statistics… button on the right side of the dialog box.
Check the box labeled Kappa.
Click Continue.
5. Run and Interpret
Click OK in the main dialog box. SPSS will generate an Output window with a table named "Symmetric Measures". Look for the row labeled "Kappa". The "Value" column indicates your Cohen's Kappa coefficient, and the "Approx. Sig." tells you if the agreement is statistically significant.
Interpreting Your Results
Once you have your Kappa value (either from our calculator or SPSS), you can interpret the strength of agreement using the standard Landis and Koch (1977) scale:
Kappa Value
Strength of Agreement
< 0.00
Poor
0.00 – 0.20
Slight
0.21 – 0.40
Fair
0.41 – 0.60
Moderate
0.61 – 0.80
Substantial
0.81 – 1.00
Almost Perfect
Common Issues in SPSS
"No measures of association are computed": This usually happens if one rater gave the same score for every single item (zero variance). Kappa cannot be calculated if a rater has a constant value because the denominator in the formula becomes zero.
function calculateKappa() {
// 1. Get Inputs
var a = parseFloat(document.getElementById('valA').value);
var b = parseFloat(document.getElementById('valB').value);
var c = parseFloat(document.getElementById('valC').value);
var d = parseFloat(document.getElementById('valD').value);
// Validation: Check if inputs are numbers
if (isNaN(a)) a = 0;
if (isNaN(b)) b = 0;
if (isNaN(c)) c = 0;
if (isNaN(d)) d = 0;
// 2. Calculate Totals
var total = a + b + c + d;
// Handle edge case of no data
if (total === 0) {
alert("Please enter at least one observation.");
return;
}
// Marginal Totals
var rater1Yes = a + b;
var rater1No = c + d;
var rater2Yes = a + c;
var rater2No = b + d;
// 3. Observed Agreement (Po)
// (Agreed Yes + Agreed No) / Total
var observedAgreement = (a + d) / total;
// 4. Expected Agreement (Pe)
// Probability of random agreement
var probYes = (rater1Yes / total) * (rater2Yes / total);
var probNo = (rater1No / total) * (rater2No / total);
var expectedAgreement = probYes + probNo;
// 5. Calculate Kappa
// K = (Po – Pe) / (1 – Pe)
var kappa = 0;
// Prevent division by zero if Pe is 1 (Perfect expected agreement)
if (expectedAgreement === 1) {
kappa = 0; // Usually treated as 0 or undefined in this edge case
} else {
kappa = (observedAgreement – expectedAgreement) / (1 – expectedAgreement);
}
// 6. Interpretation Logic (Landis & Koch)
var interpretationText = "";
var boxColor = "#6c757d"; // default gray
if (kappa < 0) {
interpretationText = "Poor Agreement (Less than chance)";
boxColor = "#dc3545"; // Red
} else if (kappa <= 0.20) {
interpretationText = "Slight Agreement";
boxColor = "#e67e22"; // Orange
} else if (kappa <= 0.40) {
interpretationText = "Fair Agreement";
boxColor = "#f1c40f"; // Yellow
} else if (kappa <= 0.60) {
interpretationText = "Moderate Agreement";
boxColor = "#17a2b8"; // Cyan
} else if (kappa <= 0.80) {
interpretationText = "Substantial Agreement";
boxColor = "#28a745"; // Green
} else {
interpretationText = "Almost Perfect Agreement";
boxColor = "#155724"; // Dark Green
}
// 7. Update DOM
document.getElementById('resTotal').innerText = total;
document.getElementById('resPo').innerText = (observedAgreement * 100).toFixed(2) + "%";
document.getElementById('resPe').innerText = (expectedAgreement * 100).toFixed(2) + "%";
document.getElementById('resKappa').innerText = kappa.toFixed(3);
var interpretBox = document.getElementById('interpretation');
interpretBox.innerText = interpretationText;
interpretBox.style.backgroundColor = boxColor;
// Show results
document.getElementById('results').style.display = 'block';
}