Comparison of ROC Curves

A new biomarker rarely enters clinical practice in isolation. It is almost always evaluated against an established test — a higher-cost assay, a more invasive procedure, or the current standard of care. The question is not merely whether the new test is good in absolute terms; it is whether it is $\mathrm{better\; than\; what\; already\; exists}$. Answering that question rigorously requires a formal statistical comparison of the two tests' ROC curves, and specifically of their areas under the curve.

Visual inspection is not sufficient. Two ROC curves may cross, or one may appear to dominate only over a limited range of the false positive rate. Even when one AUC is numerically larger than the other, the difference could easily arise from sampling variability if the study is small. A statistical test is needed to decide whether the observed difference is greater than what would be expected by chance.

Two Study Designs: Paired and Independent Samples

The appropriate statistical method depends on how the study was conducted. There are two fundamentally different designs:

• Paired (dependent) samples: Both tests are performed on the same patients. For example, two serum biomarkers — CRP and procalcitonin — are each measured in the same group of 120 patients with suspected sepsis. The two ROC curves are derived from a single cohort.
• Independent samples: The two tests are each evaluated in separate, non-overlapping groups of patients. This design is less efficient and less common in head-to-head diagnostic accuracy studies, but arises when historical datasets from different studies are being compared, or when ethical or practical constraints prevent measuring both tests in the same patients.

The distinction is critical because it determines whether the correlation between the two AUC estimates must be accounted for. In a paired study, patients who are intrinsically "difficult to classify" will score ambiguously on $\mathrm{both}$ tests. This creates a positive correlation between the two AUCs — a correlation that, if ignored, leads to an inflated standard error of the difference, reduced power, and a conservative test. Handling the two designs with the same formula is a methodological error.

The DeLong Method

The standard and recommended method for comparing two AUCs — whether from paired or independent samples — is that of DeLong, DeLong and Clarke-Pearson (1988). It is a fully non-parametric approach: it makes no assumption about the distribution of test values in the diseased or healthy population.

The starting point is the recognition that the AUC is mathematically equivalent to the Wilcoxon rank-sum statistic. For a test with $n_1$ diseased patients (cases) and $n_0$ healthy patients (controls), the AUC can be written as:

where the kernel function $\psi(X_1, X_0)$ equals 1 if the case value exceeds the control value, 0.5 if they are tied, and 0 if the control value exceeds the case value. This is simply the probability that a randomly selected case scores higher than a randomly selected control — the standard probabilistic interpretation of the AUC.

From this representation, DeLong et al. derived a closed-form estimator for the variance of $\hat{\theta}$ using the theory of structural components (a generalization of the jackknife). This variance estimator is used to build both the confidence interval for a single AUC and, in the case of two tests, the covariance between their AUC estimates.

Paired Comparison

When both tests are measured on the same patients, the comparison uses:

The covariance term $\text{Cov}(\hat{\theta}_1, \hat{\theta}_2)$ captures how much the two AUC estimates move together across the shared sample. DeLong et al. show that this covariance can be estimated directly from the structural components of both tests evaluated on the same cases and controls, without needing to enumerate all possible resamplings. The denominator is thus the standard error of the $\mathrm{difference}$, correctly adjusted for the dependence between the two tests.

Under the null hypothesis that the two tests have equal discriminative ability ($\theta_1 = \theta_2$), the statistic $z$ follows a standard normal distribution. The two-sided p-value is $2 \times \Phi(-|z|)$, where $\Phi$ is the standard normal CDF. A 95% confidence interval for the true difference $\theta_1 - \theta_2$ is:

Why the covariance matters: Consider two biomarkers both measured in 100 patients with and without myocardial infarction. The AUC for Biomarker A is 0.82 and for Biomarker B is 0.78, a difference of 0.04. If the two biomarkers are highly correlated (patients with equivocal troponin levels also tend to have equivocal BNP levels), the covariance term is large and positive, making the standard error of the difference small — the test has power to detect even a modest true difference. If you naively treated the two AUCs as independent, you would overestimate the standard error, understate the z-statistic, and potentially fail to detect a real superiority.

Independent Comparison

When the two tests are measured in separate, non-overlapping patient groups, the samples are independent, so $\text{Cov}(\hat{\theta}_1, \hat{\theta}_2) = 0$ by definition. The test statistic simplifies to:

Each variance is estimated by the DeLong method applied to its own dataset. Everything else — the normal approximation, the p-value, the confidence interval — proceeds identically to the paired case.

Comparing Three or More ROC Curves

When a study evaluates $k \geq 3$ tests in the same patients, a global hypothesis test is performed before any pairwise comparisons. The global null hypothesis is that all $k$ tests have equal AUC: $\theta_1 = \theta_2 = \cdots = \theta_k$.

The DeLong framework extends naturally to this setting. The $k$ AUC estimates and their full $k \times k$ covariance matrix are computed from the structural components. A set of $k - 1$ independent contrasts (e.g., $\theta_1 - \theta_2,\; \theta_1 - \theta_3, \ldots$) is assembled into a vector $\mathbf{c}$, and the global test statistic is:

where $\mathbf{V}$ is the estimated covariance matrix of the contrasts. Under the global null, this statistic follows a chi-square distribution with $k - 1$ degrees of freedom. If the global test is significant, pairwise comparisons of individual AUC pairs can be performed — each using the z-test described above — to determine which tests differ from which. MedCalc performs all $\binom{k}{2}$ pairwise comparisons automatically and reports the result for each pair.

Example: A study evaluates three urinary biomarkers — NGAL, KIM-1, and cystatin C — for early detection of acute kidney injury in the same 200 patients. The global chi-square test (df = 2) first assesses whether there is $\mathrm{any}$ difference among the three AUCs (0.84, 0.79, 0.76). If p < 0.05, pairwise comparisons are reported: NGAL vs. KIM-1, NGAL vs. cystatin C, and KIM-1 vs. cystatin C. Only pairwise comparisons that survive this step should be interpreted as formally established.

The Hanley & McNeil Alternative

Before the DeLong method, Hanley and McNeil (1983) proposed a method for paired comparisons that estimates the correlation between two AUCs from a table based on the average AUC value and the observed correlation $r$ between the two test scores across all subjects. Their formula is:

where $r'$ is a tabulated approximation to the true AUC correlation, derived from the average of $\hat{\theta}_1$ and $\hat{\theta}_2$ and from the Pearson correlation $r$ between the raw test scores. The DeLong method is preferred because it does not rely on this approximation — it estimates the covariance directly from the data, without requiring the raw-score correlation as an intermediate step. The Hanley–McNeil approach is retained in MedCalc as an option for replication of older published analyses.

Statistical Significance vs. Clinical Relevance

A statistically significant difference in AUC does not automatically translate into a clinically meaningful one. With a large enough sample, even a difference of 0.01 between two AUCs can reach p < 0.05. Whether such a difference justifies replacing one test with another depends on considerations that lie outside the statistical framework:

• Absolute magnitude of the difference: A difference of 0.03 in AUC (e.g., 0.85 vs. 0.82) may have negligible clinical consequences in most settings. A difference of 0.10 or more is generally regarded as substantively important.
• Location of the difference on the ROC curve: If the two curves cross, one test may dominate in the high-sensitivity region (relevant for screening) and the other in the high-specificity region (relevant for confirmation). The overall AUC comparison can obscure this distinction; visual inspection of the curves remains essential.
• Cost, availability, and harm: If two tests have statistically indistinguishable AUCs, the clinically superior choice is the one that is cheaper, faster, or less invasive — considerations the AUC cannot capture.
• Confidence interval for the difference: The 95% CI for $\hat{\theta}_1 - \hat{\theta}_2$ should always be examined alongside the p-value. A wide interval spanning zero means the data are consistent with both a meaningful advantage and a meaningful disadvantage for the new test, and the study was likely underpowered. A narrow interval straddling zero gives a more informative verdict: the two tests are genuinely similar.

Reporting guidance: When comparing ROC curves, report the AUC and its 95% confidence interval for each test, the difference in AUC with its 95% confidence interval, and the p-value for the comparison. The confidence interval for the difference is more informative than the p-value alone: it shows both the direction and the plausible range of any advantage.

MedCalc Procedures

MedCalc implements both study designs as separate procedures:

• Comparison of ROC curves — for paired (dependent) samples: 2 to 6 tests measured on the same patients. Reports pairwise differences in AUC with standard errors, 95% confidence intervals, and p-values; performs the global chi-square test when more than two curves are compared. Both the DeLong and Hanley & McNeil methods are available.
• Comparison of independent ROC curves — for independent (unpaired) samples: two tests evaluated in separate patient groups. Reports the difference in AUC with standard error, 95% confidence interval, and p-value using the DeLong or Hanley & McNeil method.