Use Inter-rater agreement to evaluate the agreement between two classifications (nominal or ordinal scales).
Agreement is quantified by the Kappa (K) statistic:
In the dialog form you can enter the two classification systems in a 6x6 frequency table.
Select Weighted Kappa if the data come from an ordered scale. If the data come from a nominal scale, do not select Weighted Kappa.
Use linear weights when the difference between the first and second category has the same importance as a difference between the second and third category, etc. If the difference between the first and second category is less important than a difference between the second and third category, etc., use quadratic weights.
In this example, from the 6 cases that observer B has placed in class 1, observer A has placed 5 in class 1 and 1 in class 2; from the 19 cases that observer B has placed in class 2, observer A has placed 3 in class 1, 12 in class 2 and 4 in class 3; and from the 12 cases that observer B has placed in class 3, observer A has placed 2 in class 1, 2 in class 2 and 8 in class 3.
After you have entered the data, click the Test button. The program will display the value for Kappa with its Standard Error and 95% confidence interval (CI) (Fleiss et al., 2003).
The Standard errors reported by MedCalc are the appropriate standard errors for testing the hypothesis that the underlying value of weighted kappa is equal to a prespecified value other than zero (Fleiss, 2003).
The K value can be interpreted as follows (Altman, 1991):
In the Comment input field you can enter a comment or conclusion that will be included on the printed report.