P-values and Confidence Intervals
The $p$-value and the confidence interval are the two most widely reported quantities in clinical research, and the two most widely misunderstood. They are not interchangeable summaries of the same thing: a $p$-value is a test of a specific hypothesis, while a confidence interval is an estimate of a quantity with a measure of its uncertainty. Used together, they provide a more complete picture than either alone. This chapter explains what each is, how each is calculated, and how the two relate to each other. The question of how to interpret them in clinical context — and the distinction between statistical and clinical significance — is taken up in the Clinical versus statistical significance chapter.
Hypothesis Testing
Most statistical tests begin with a precisely stated null hypothesis $H_0$ — a default claim that there is no effect, no difference, or no association. Examples include "the mean blood pressure is the same in both treatment groups", "the correlation between age and creatinine is zero", or "the two proportions are equal". The null hypothesis is not the hypothesis of interest; it is the foil against which the data are compared.
A test statistic is then computed from the data. Its exact form depends on the type of data and the null hypothesis being tested, but in every case it summarises how far the observed data deviate from what the null hypothesis predicts. A $t$-statistic compares an observed mean difference to the variability expected under $H_0$; a $\chi^2$ statistic compares observed cell counts to expected counts; a $z$-statistic compares an observed proportion or correlation to its null value. The larger the test statistic in absolute value, the more the data depart from the null hypothesis.
To convert the test statistic into a probability, it is referred to its sampling distribution under $H_0$ — the theoretical distribution the statistic would follow if $H_0$ were true and the study were repeated many times with new samples of the same size. The area in the tail of this distribution beyond the observed statistic is the $p$-value.
The P-value
The $p$-value is the probability of obtaining a test statistic at least as extreme as the one observed, assuming the null hypothesis is true. For a two-tailed test — which tests for a departure from $H_0$ in either direction — "at least as extreme" means at least as far from the null value in either direction, and the $p$-value is the combined area in both tails.
A small $p$-value means that the observed data would be unlikely if $H_0$ were true: either something rare has happened by chance, or $H_0$ is false. By convention, $p < 0.05$ is taken as the threshold for declaring a result "statistically significant" — meaning the probability of observing data this extreme, if there were truly no effect, is less than 5%. This threshold is arbitrary, as discussed in the Clinical versus statistical significance chapter.
The Confidence Interval
While the $p$-value tests a hypothesis, the confidence interval (CI) estimates a quantity — a mean, a difference, a proportion, an odds ratio — together with a measure of how precisely it has been determined. A 95% CI is constructed so that, in the long run, 95% of such intervals calculated from repeated samples of the same size would contain the true population value.
The width of the confidence interval is determined by two factors: the variability in the data (larger SD → wider interval) and the sample size (larger $n$ → narrower interval). A narrow CI means the estimate is precise; a wide CI means considerable uncertainty remains about the true value. This is information the $p$-value does not carry.
The Relationship Between P-values and Confidence Intervals
There is a mathematical equivalence between $p$-values and confidence intervals for the same hypothesis test. A 95% CI that the null value (typically 0 for a difference, or 1 for a ratio) usually corresponds to $p < 0.05$ in the two-tailed test. A 95% CI that the null value corresponds to $p \geq 0.05$.
Because they carry equivalent information about statistical significance, neither is superior to the other on that dimension. The confidence interval is, however, strictly more informative overall: it communicates the of the effect, its , and the of the estimate — all of which the $p$-value discards in favour of a single number. Knowing that a treatment reduces systolic blood pressure with $p = 0.03$ tells you the result is significant; knowing the 95% CI is −12 to −2 mmHg tells you the same thing and also that the effect is somewhere between clinically marginal and clinically meaningful, with the true value more likely at the larger end. Both pieces of information are needed for clinical interpretation.
For this reason, reporting guidelines for clinical research — including CONSORT for trials and STROBE for observational studies — require that results be reported as effect estimates with confidence intervals, not as $p$-values alone.
Common Misconceptions
Four misinterpretations of $p$-values and confidence intervals recur so frequently in the medical literature that they deserve explicit correction.
The $p$-value is not the probability that the null hypothesis is true. A $p$-value of 0.03 does not mean there is a 3% chance the null hypothesis is correct. It means that if the null hypothesis were correct, data as extreme as those observed would arise 3% of the time. These are different statements. The probability that $H_0$ is true depends on prior evidence, which the $p$-value does not incorporate.
The confidence interval does not describe where 95% of individual observations fall. That would be a reference interval or prediction interval, not a confidence interval. The 95% CI describes where the (the true mean, difference, or ratio) is likely to lie — not individual data points. A 95% CI for a mean blood pressure difference of −7 mmHg could be very narrow even though individual patients vary widely.
A non-significant result does not mean no effect exists. A result with $p = 0.12$ is consistent with a genuine effect that the study was too small to detect reliably, or with no effect at all. These possibilities cannot be distinguished from the $p$-value alone. The width of the confidence interval is the right tool: a narrow interval that brackets only small, clinically unimportant values is informative evidence of absence; a wide interval that includes both clinically important and trivial values is simply uninformative.
The "95% confidence" is a statement about the procedure, not the interval. Once an interval has been calculated from a specific dataset, the true parameter is either inside it or not — there is no probability involved. The 95% refers to the long-run performance of the method: 95% of intervals produced by the same procedure from repeated samples would contain the true value. This distinction matters most in small samples and near boundaries, and is a reason some statisticians prefer Bayesian credible intervals, which do carry a direct probability statement about the parameter.