Intraclass Correlation Coefficient
The intraclass correlation coefficient (ICC) quantifies the reliability of continuous measurements obtained by two or more raters, observers, or instruments applied to the same subjects. It answers the question: what proportion of the total variability in the measurements is attributable to true differences between subjects, rather than to measurement error or rater variability? An ICC near 1 indicates that nearly all observed variation reflects genuine between-subject differences — the measurements are highly reliable and interchangeable. An ICC near 0 indicates that the measurements are dominated by error and carry little information about the true underlying quantity. The ICC is widely used in clinical research for assessing inter-rater reliability (do two clinicians score the same patient consistently?), intra-rater reliability (does the same clinician score a patient consistently on two occasions?), and instrument reliability. Unlike the Bland–Altman plot, which visualises the pattern of disagreement, the ICC summarises reliability as a single index suitable for reporting and for power calculations in reliability studies.
The ICC as a Variance Ratio
The ICC is defined as the proportion of total measurement variance that is due to differences between subjects:
$$ \text{ICC} = \frac{\sigma^2_{\text{subjects}}}{\sigma^2_{\text{subjects}} + \sigma^2_{\text{error}}} $$When measurements are highly reliable, $\sigma^2_{\text{subjects}}$ is large relative to $\sigma^2_{\text{error}}$ — raters agree closely and observed differences between subjects mainly reflect true biological variation. When measurements are unreliable, $\sigma^2_{\text{error}}$ dominates and the ICC is low. In practice, these variance components are estimated from the mean squares of a one-way or two-way ANOVA applied to the reliability data.
Choosing a Model
Three design questions determine which ICC model is appropriate:
Are the same raters used for every subject? If each subject is rated by a randomly selected subset of raters from a larger pool — so that the particular raters who assessed one subject are not the same as those who assessed another — the one-way random-effects model is appropriate. This model treats all rater-related variability as part of the error, and it can only measure absolute agreement. It is the most conservative of the models and is rarely the right choice for typical clinical reliability studies, where the same raters assess all subjects.
When the raters assess every subject — the usual situation in practice — a two-way model is used, which separates rater-to-rater variability from residual error. The two-way model comes in two variants: two-way random, when the raters in the study were themselves randomly sampled from a broader population of raters and the results are intended to generalise to that population; and two-way mixed, when the raters are specifically the ones of interest and results are not intended to generalise beyond them. In practice, two-way mixed is the more commonly justified choice: the raters are usually the specific individuals or instruments under investigation.
Does systematic rater bias matter? This is the choice between absolute agreement and consistency, discussed in the next section.
Single or average measures? This is discussed in the section after that.
Consistency vs Absolute Agreement
Consider three subjects rated by two observers: the pairs of ratings are (2, 4), (4, 6), and (6, 8). Observer B scores exactly 2 points higher than Observer A in every case. The rankings are perfectly preserved — a subject rated highest by A is also rated highest by B — but there is a constant systematic bias of 2 units between the raters.
Consistency asks whether the raters rank the subjects in the same order and by the same relative distances, ignoring any systematic offset between them. In this example, consistency is perfect: ICC = 1.0. Consistency is the appropriate choice when the systematic difference between raters is irrelevant to the purpose of the measurement — for instance, when two instruments both measure the same physiological quantity on different scales, or when a rater's overall level does not matter because results from different raters will never be directly compared.
Absolute agreement asks whether raters produce numerically identical values. The systematic offset of 2 units in the example above is penalised, and the ICC for absolute agreement is 0.67. Absolute agreement is the appropriate choice when measurements from different raters are used interchangeably — for example, when a patient may be assessed by any member of a clinical team and decisions depend on the actual value, not merely its rank.
The one-way random model always produces absolute agreement. The two-way models allow both; the choice should be guided by how the measurements will be used.
Single vs Average Measures
The single-measures ICC is the reliability index for one typical rater: it answers the question "how reliable is any single measurement taken by one rater?" The average-measures ICC is the reliability of the of $k$ raters' measurements, and is always higher than the single-measures ICC. The relationship between the two follows the Spearman–Brown formula: averaging over $k$ raters reduces the contribution of random error by a factor of $k$, which raises the ICC.
For most clinical purposes — a single clinician, a single instrument reading — the single-measures ICC is the relevant quantity. The average-measures ICC is appropriate when the standard protocol is to take multiple measurements and report their mean.
Interpreting the ICC
| ICC | Reliability |
|---|---|
| < 0.50 | Poor |
| 0.50 – 0.75 | Moderate |
| 0.75 – 0.90 | Good |
| > 0.90 | Excellent |
These thresholds should be applied with caution. The ICC is sensitive to the range of subjects in the study: a sample drawn from a population with wide biological variation will yield a higher ICC than the same raters and instruments applied to a more homogeneous sample, even if measurement error is identical in both cases. A reliability study should therefore enrol subjects representative of the full clinical spectrum over which the measurement will be used.
Worked Example
Two cardiologists independently measure left ventricular ejection fraction (LVEF, %) by echocardiography in the same 10 patients:
| Patient | Cardiologist A | Cardiologist B |
|---|---|---|
| 1 | 55 | 58 |
| 2 | 42 | 44 |
| 3 | 65 | 67 |
| 4 | 38 | 40 |
| 5 | 72 | 75 |
| 6 | 48 | 50 |
| 7 | 60 | 63 |
| 8 | 35 | 37 |
| 9 | 68 | 71 |
| 10 | 52 | 54 |
Cardiologist B measures consistently 2–3 percentage points higher than Cardiologist A across all patients. The rankings are perfectly preserved. Because both cardiologists always assess the same patients and results will be used interchangeably in clinical reports, a two-way mixed model is appropriate. The choice between consistency and absolute agreement depends on clinical context: if a treatment threshold (for example, LVEF < 40%) will be applied to any individual measurement regardless of which cardiologist performed it, absolute agreement is the relevant measure.
Calculating the ICC for this data yields the following results:
| Single measures | Average measures (k = 2) | |
|---|---|---|
| Consistency | 0.9992 | 0.9996 |
| Absolute agreement | 0.9825 | 0.9912 |
All four values are in the range. Consistency is near-perfect (0.9992) because the raters rank patients identically with negligible residual scatter. Absolute agreement is slightly lower (0.9825) because the systematic 2–3 point offset between raters is penalised — a patient measured at LVEF 38% by Cardiologist A might be reported as 40% by Cardiologist B, a difference that matters clinically near a decision threshold. For routine clinical use, absolute agreement is the more demanding and appropriate standard here; its value of 0.9825 still indicates excellent interchangeability.
MedCalc computes the intraclass correlation coefficient for all model and agreement combinations, with 95% confidence intervals, in its Intraclass correlation coefficient procedure.