Inter-Rater Agreement (Kappa)
When two observers independently classify the same subjects into categories — diagnosing a lesion as benign or malignant, staging a tumour, or rating pain severity on an ordinal scale — simple percentage agreement is a misleading measure of concordance because some agreement will occur by chance even if both raters are guessing randomly. Cohen's kappa corrects for this by expressing agreement as a proportion of the agreement that remains after chance agreement is removed. A kappa of 1 indicates perfect agreement; a kappa of 0 indicates only chance agreement; negative values indicate agreement worse than chance. For ordinal classifications, where disagreements of one category apart are less serious than disagreements of several categories, weighted kappa assigns partial credit to near-misses according to a weighting scheme, giving a more nuanced picture of rater concordance. Kappa statistics are reported routinely in studies validating diagnostic criteria, rating scales, and clinical classification systems.
Cohen's Kappa
Let $P_o$ be the observed agreement — the proportion of subjects on which the two raters agree — and $P_e$ the expected agreement — the proportion that would agree by chance alone given the marginal totals. Cohen's kappa is:
$$ \kappa = \frac{P_o - P_e}{1 - P_e} $$The denominator $1 - P_e$ is the maximum possible improvement over chance; the numerator $P_o - P_e$ is the actual improvement. Kappa is therefore the fraction of the available beyond-chance agreement that was actually achieved.
Expected agreement is calculated from the marginal totals of the agreement table. For each category $i$, the expected proportion of agreements is the product of the proportion of subjects that rater A placed in category $i$ and the proportion that rater B placed in category $i$. Summing these products over all categories gives $P_e$. The calculation is the same in principle as computing expected cell counts in the chi-squared test.
A key limitation of unweighted kappa is that all disagreements are treated identically, regardless of their magnitude. Rating a patient as Grade 1 when the correct grade is Grade 2 counts as the same disagreement as rating Grade 1 when the correct grade is Grade 4. For nominal classifications — where categories have no natural ordering — this is appropriate. For ordinal classifications, weighted kappa is more appropriate.
Weighted Kappa
Weighted kappa extends the basic formula to ordinal scales by assigning a weight to each cell of the agreement table based on how far apart the two ratings are. A weight of 1 is assigned to perfect agreement (cells on the diagonal); smaller weights are assigned to larger discrepancies; a weight of 0 is assigned to the maximum possible disagreement.
Two weighting schemes are in common use. For $k$ ordered categories, the distance between two ratings that are $i$ categories apart ($i = 0, 1, \ldots, k-1$) is weighted as follows:
Linear weights:
$$ w_i = 1 - \frac{i}{k - 1} $$Quadratic weights:
$$ w_i = 1 - \frac{i^2}{(k - 1)^2} $$With linear weights, the penalty for disagreement increases proportionally with the number of categories separating the two ratings — a one-step disagreement and a two-step disagreement are penalised proportionally. Quadratic weights penalise large discrepancies more severely than small ones: a one-step disagreement receives more credit than under linear weighting, but a disagreement spanning nearly all categories is penalised more heavily. Quadratic weights are equivalent to the intraclass correlation coefficient when data are ordinal with equally spaced categories, and are generally preferred for Likert-type scales.
When the outcome has only two categories, weighted and unweighted kappa are identical.
Interpreting Kappa
The following benchmarks, widely adopted in the clinical literature, provide a rough guide to interpreting kappa values:
| Kappa | Strength of agreement |
|---|---|
| < 0.20 | Poor |
| 0.21 – 0.40 | Fair |
| 0.41 – 0.60 | Moderate |
| 0.61 – 0.80 | Good |
| 0.81 – 1.00 | Very good |
These thresholds should be interpreted in context. A kappa of 0.65 may be entirely acceptable for a complex subjective staging system and wholly inadequate for a binary laboratory test. The magnitude of kappa also depends on the prevalence of each category: when one category is very common, raters will agree on it frequently by chance, and the correction for chance agreement can make kappa appear deceptively low even when practical concordance is high. Reporting both percentage agreement and kappa together helps readers judge this.
Worked Example
Two pathologists independently grade 60 tumour biopsy specimens using a three-tier system: Grade 1 (well differentiated), Grade 2 (moderately differentiated), Grade 3 (poorly differentiated). The results are:
| Path B: Grade 1 | Path B: Grade 2 | Path B: Grade 3 | Total | |
|---|---|---|---|---|
| Path A: Grade 1 | 22 | 4 | 0 | 26 |
| Path A: Grade 2 | 3 | 18 | 3 | 24 |
| Path A: Grade 3 | 0 | 2 | 8 | 10 |
| Total | 25 | 24 | 11 | 60 |
The two pathologists agree on 48 of 60 specimens (22 + 18 + 8), giving an observed agreement of $P_o = 48/60 = 0.800$. Expected agreement, calculated from the marginal totals, is $P_e = 0.371$. Cohen's kappa is therefore:
$$ \kappa = \frac{0.800 - 0.371}{1 - 0.371} = \frac{0.429}{0.629} = 0.68 $$This falls in the range. However, most disagreements are between adjacent grades: three specimens were rated Grade 1 by one pathologist and Grade 2 by the other, and five were graded one step apart in the other direction. Only one pair of ratings spans two grades (Grade 1 vs Grade 3 does not occur at all in this dataset). Unweighted kappa treats all off-diagonal cells equally, so it does not capture this clinically important distinction.
Applying linear weights to the three-category scale gives weights of 1 (same grade), 0.5 (one grade apart), and 0 (two grades apart). The weighted kappa is $\kappa_w = 0.74$ (95% CI 0.61 to 0.88), higher than the unweighted value. The 95% confidence interval is computed following Fleiss et al. (2003).
The reason weighted kappa exceeds unweighted kappa here is instructive. Unweighted kappa treats every off-diagonal cell as a complete failure — a one-step disagreement (Grade 1 vs Grade 2) counts the same as a two-step disagreement (Grade 1 vs Grade 3). Weighted kappa instead awards half-credit (weight = 0.5) to adjacent-grade disagreements. In this dataset, all 12 discordant specimens fall in adjacent-grade cells; no case of a two-step disagreement (Grade 1 vs Grade 3) occurs at all. Weighted kappa therefore recovers partial credit for all 12 discordant pairs, pushing the value upward. Had the disagreements instead spanned two grades — the most severe possible for a three-category scale, receiving weight 0 — weighted kappa would have been lower than unweighted, not higher.
MedCalc computes Cohen's kappa and weighted kappa (with both linear and quadratic weights) with 95% confidence intervals in its Kappa procedure.