Free statistical calculators

Description

Use Inter-rater agreement to evaluate the agreement between two classifications (nominal or ordinal scales).

Required input

• First select the number of categories in the classification system - the maximum number of categories is 12.
• Next enter the number of observations in each cell of the data table.
• Weighted Kappa: select Weighted Kappa if the data come from an ordered scale. If the data come from a nominal scale, do not select Weighted Kappa.

  MedCalc offers two sets of weights, called linear and quadratic. In the linear set, if there are k categories, the weights are calculated as follows:

  $$ w_i = 1 - \frac{i}{k-1} $$

  and in the quadratic set:

  $$ w_i = 1 - \frac{i^2}{ (k-1)^2 } $$

  When there are 5 categories, the weights in the linear set are 1, 0.75, 0.50, 0.25 and 0 when there is a difference of 0 (=total agreement) or 1, 2, 3 and 4 categories respectively. In the quadratic set the weights are 1, 0.937, 0.750, 0.437 and 0.

  Use linear weights when the difference between the first and second category has the same importance as a difference between the second and third category, etc. If the difference between the first and second category is less important than a difference between the second and third category, etc., use quadratic weights.

Interpretation

Agreement is quantified by the Kappa (K) statistic (Cohen, 1960; Fleiss et al., 2003):

• K is 1 when there is perfect agreement between the classification systems;
• K is 0 when there is no agreement better than chance;
• K is negative when agreement is worse than chance.

After you have clicked the Test button, the program will display the value for Kappa with its Standard Error and 95% confidence interval (CI) (Fleiss et al., 2003).

The Standard errors reported by MedCalc are the appropriate standard errors for testing the hypothesis that the underlying value of weighted kappa is equal to a prespecified value other than zero (Fleiss et al., 2003).

The K value can be interpreted as follows (Altman, 1991):

Literature

• Altman DG (1991) Practical statistics for medical research. London: Chapman and Hall.
• Cohen J (1960) A coefficient of agreement for nominal scales. Educational and Psychological Measurement 20:37-46.
• Fleiss JL, Levin B, Paik MC (2003) Statistical methods for rates and proportions, 3^rd ed. Hoboken: John Wiley & Sons.

How to cite this page

• MedCalc Software Ltd. Inter-rater agreement. https://www.medcalc.org/calc/kappa.php (Version 23.0.9; accessed December 8, 2024)