Skip to main content

Biostatistics in Practice: Principles and Procedures - A MedCalc companion e-book

Precision–Recall Curves

The ROC curve evaluates a diagnostic test by asking how well it separates diseased from healthy individuals at every possible threshold, and summarises the result as the AUC — a number that is independent of disease prevalence. This prevalence-independence is often presented as a strength, but in settings where the condition is rare it can become a serious problem. When the healthy population vastly outnumbers the diseased one, even a mediocre test can achieve a high AUC simply because it is easy to rank a true case above a true control when there are hundreds of controls for every case. More concretely: a test with 85% sensitivity and 95% specificity in a population where the disease affects 2% of screened individuals will have a high ROC AUC, yet its positive predictive value is only about 26% — three out of every four positive results are false alarms. The ROC framework does not surface this directly.

The precision–recall (PR) curve addresses this by replacing the false positive rate on the horizontal axis with recall (sensitivity) and replacing sensitivity on the vertical axis with precision (positive predictive value). Because precision depends directly on the ratio of true positives to all positive calls — and therefore on disease prevalence — the PR curve reflects the actual clinical burden of false positives in the population being tested. For rare conditions and for any setting where false positives are operationally costly, PR curves provide a more clinically honest picture of test performance than ROC curves.

Precision and Recall

Both metrics are defined in terms of the standard 2×2 confusion matrix. At a chosen decision threshold, the test classifies each subject as positive or negative, giving four counts: true positives (TP), false positives (FP), true negatives (TN), and false negatives (FN).

Recall is the proportion of all truly diseased subjects that the test correctly identifies as positive — identical to sensitivity:

$$ \text{Recall} = \frac{TP}{TP + FN} $$

Precision is the proportion of all subjects called positive by the test that are truly diseased — identical to the positive predictive value (PPV):

$$ \text{Precision} = \frac{TP}{TP + FP} $$

There is an inherent trade-off between the two. Lowering the decision threshold flags more subjects as positive, recovering more true cases (higher recall) but also admitting more false positives (lower precision). Raising the threshold does the opposite. The PR curve makes this trade-off explicit across the full range of thresholds.

The Precision–Recall Curve

To construct the PR curve, every observed test value is considered as a decision threshold in turn. For each threshold, recall and precision are computed and the pair is plotted as a point, with recall on the horizontal axis and precision on the vertical axis. Connecting the points traces the curve.

Precision–recall curve for an illustrative diagnostic test with 20% disease prevalence. The curve starts near (recall = 0, precision = 1) in the top-left and descends as recall increases, ending at (recall = 1, precision = 0.20) at the prevalence line. The shaded area represents the AUPRC = 0.60. A red dot marks the F1max operating point at recall = 0.60 and precision = 0.52. The dashed horizontal line at precision = 0.20 is the no-skill baseline.

The curve moves from left to right as the threshold is lowered. At the most restrictive threshold (top-left), only the single highest-confidence subject is called positive. If that subject is truly diseased, precision is 1.0 and recall is tiny. As the threshold falls, recall rises — more true cases are identified — but precision generally falls, as false positives accumulate. The curve typically ends at the right edge where recall reaches 1.0 (every diseased subject is captured) and precision equals the disease prevalence (because the test is now calling everyone positive).

The No-Skill Baseline

A critical difference between the PR and ROC frameworks is the shape of the no-skill baseline — the performance expected from a classifier that assigns labels randomly.

The practical consequence is that even a modest AUPRC can represent a substantial improvement over the baseline when the condition is rare. An AUPRC of 0.30 against a baseline of 0.02 represents a 15-fold lift in precision over random labelling — a clinically important achievement. The same AUPRC of 0.30 against a baseline of 0.25 (25% prevalence) represents very little gain. Interpreting AUPRC without reference to the prevalence baseline is therefore meaningless.

Area Under the Precision–Recall Curve

The AUPRC summarises the entire PR curve as a single number, analogous to the AUC for ROC curves. It ranges from the prevalence (no-skill baseline) to 1.0 (perfect classifier). A higher AUPRC indicates that the test maintains high precision across a broad range of recall levels.

A subtlety arises in computing the AUPRC: simple linear interpolation between adjacent points on the curve substantially overestimates the true area, because the relationship between precision and recall as the threshold changes is non-linear. MedCalc uses the non-linear interpolation method of Davis & Goadrich (2006), which correctly accounts for the number of positive and negative examples between consecutive threshold values.

Confidence Interval for the AUPRC

The 95% confidence interval for the AUPRC uses the logit method (Boyd et al., 2013). The AUPRC is first transformed to an unbounded scale via the logit function:

$$ \mu_\eta = \ln\!\left(\frac{\text{AUPRC}}{1 - \text{AUPRC}}\right) $$

The standard error on this transformed scale is approximated as:

$$ \tau = \frac{1}{\sqrt{n \times \text{AUPRC} \times (1 - \text{AUPRC})}} $$

where $n$ is the total number of subjects. A symmetric 95% interval $[\mu_\eta - 1.96\tau,\; \mu_\eta + 1.96\tau]$ is constructed on the logit scale and then mapped back through the inverse logit to give the CI for the AUPRC itself:

$$ \left[\frac{e^{\mu_\eta - 1.96\tau}}{1 + e^{\mu_\eta - 1.96\tau}},\quad \frac{e^{\mu_\eta + 1.96\tau}}{1 + e^{\mu_\eta + 1.96\tau}}\right] $$

If the lower bound of this CI exceeds the prevalence, the test performs significantly better than the no-skill baseline.

The F1 Score

While the AUPRC summarises performance across all thresholds, clinical use requires choosing a single operating point. The F1 score is the harmonic mean of precision and recall at a given threshold:

$$ F_1 = 2 \times \frac{\text{Precision} \times \text{Recall}}{\text{Precision} + \text{Recall}} = \frac{2\,TP}{2\,TP + FP + FN} $$

The harmonic mean is used rather than the arithmetic mean because it penalises extreme imbalance between precision and recall — a test that achieves precision = 1.0 and recall = 0.01 has an arithmetic mean of 0.505, suggesting good average performance, while the harmonic mean correctly gives F1 = 0.02, reflecting that the test identifies almost no cases in practice.

The threshold that maximises F1 across all possible values — denoted F1max — is the natural operating point when precision and recall are considered equally important. MedCalc reports F1max alongside the corresponding threshold value, recall, and precision. In the example above (prevalence 20%), the F1max of 0.56 is achieved at recall = 0.60 and precision = 0.52.

When F1 and Youden diverge: In a rare-disease screening programme the Youden index (which treats false positives and false negatives symmetrically in ROC space, regardless of prevalence) may recommend a threshold that floods the follow-up pipeline with false positives. The F1max operating point explicitly balances precision and recall and is therefore more practically relevant in low-prevalence settings.

PR Curves vs. ROC Curves: When to Use Which

The two frameworks are complementary, not competing. Each answers a different question:

Davis & Goadrich (2006) showed formally that a classifier with a higher ROC AUC does not necessarily have a higher AUPRC — the rankings can differ, and for imbalanced datasets the AUPRC ranking is the more meaningful one. Saito & Rehmsmeier (2015) demonstrated with simulated data that ROC curves can give an "optimistic" picture of classifier performance when the negative class vastly outnumbers the positive class, while PR curves correctly reflect the difficulty of the classification problem.

Comparing Two PR Curves

When two tests are evaluated in the same study, it is often of interest to determine whether their AUPRCs differ significantly. As with the comparison of ROC curves, the design may be paired (both tests measured in the same subjects) or independent (tests measured in separate groups).

Unlike the ROC comparison, no closed-form variance estimator for the AUPRC exists (there is no PR-curve equivalent of the DeLong method). MedCalc therefore uses the BCa (bias-corrected and accelerated) bootstrap (Efron, 1987) throughout: the dataset is resampled with replacement many times, the difference in AUPRC is recalculated for each replicate, and the 2.5th and 97.5th percentiles of the bootstrap distribution form the 95% CI for the difference. The BCa correction adjusts for both bias in the bootstrap distribution and for the skewness of the sampling distribution, giving better coverage than the simpler percentile bootstrap when sample sizes are moderate.

For a paired design, cases and controls are resampled jointly so that the correlation structure between the two test measurements is preserved. For an independent design, the two groups are resampled separately. If the 95% BCa CI for the difference in AUPRC excludes zero, the difference is statistically significant at $p < 0.05$.

Example: Two lateral-flow assays for a rare fungal infection (prevalence 4%) are each tested in the same 600 patients. Assay A achieves AUPRC = 0.58 (95% CI 0.49–0.66) and Assay B achieves AUPRC = 0.44 (95% CI 0.36–0.53). The 95% BCa bootstrap CI for the difference (0.14; 95% CI 0.03–0.25) excludes zero. At a prevalence where the no-skill baseline is 0.04, the additional 0.14 in AUPRC represents a practically meaningful superiority of Assay A, directing more true positives to treatment while generating fewer false-positive referrals.

MedCalc implements precision–recall analysis in two procedures: