Multiple Regression

The Regression chapter fits a line or curve relating a single predictor to an outcome. Clinical questions rarely stay that simple: a patient's LDL cholesterol depends on age, body mass index, diet, and genetics simultaneously, and a treatment's apparent effect on an outcome can be distorted by imbalances in age, disease severity, or other variables that differ between comparison groups. Multiple regression extends simple linear regression to two or more predictors, fitted together in a single equation:

The coefficients are again estimated by least squares — minimising the sum of squared residuals — but the fitted equation now describes a plane (with two predictors) or a hyperplane (with more), rather than a line. Multiple regression serves two distinct purposes in clinical research, and it is worth being clear about which one a given analysis is for: prediction, where the goal is an accurate estimate of $y$ from a combination of predictors, and adjustment, where the goal is to estimate the association between one particular predictor and $y$ while holding the others constant — the statistical version of controlling for confounding variables. The two goals call for different model-building strategies, discussed below. For a binary outcome, the analogous method is logistic regression, which models the log odds of the outcome rather than its raw value but shares the same logic of fitting several predictors together.

Assessing Model Fit

The coefficient of determination $R^2$ extends directly from simple regression: it is the proportion of the variation in $y$ explained by the model, calculated from the same ratio of explained to total variation described in the Regression chapter. With multiple predictors, $R^2$ has an awkward property: it can only increase, never decrease, as more predictors are added — even predictors that are pure noise. This makes raw $R^2$ unsuitable for comparing models with different numbers of predictors, or for deciding whether an additional variable is worth keeping.

The adjusted $R^2$ corrects for this by penalising the addition of predictors that do not improve the fit enough to justify the loss of degrees of freedom:

where $n$ is the sample size and $k$ is the number of predictors. Unlike $R^2$, adjusted $R^2$ can decrease when a predictor is added that contributes little, which makes it the more honest measure for comparing models of different sizes.

The residual standard deviation $s_{\text{res}}$ — the standard deviation of the differences between observed and fitted values — expresses the typical size of a prediction error in the original units of $y$:

An analysis of variance $F$-test assesses whether the model as a whole explains a significant proportion of the variation in $y$; a small $P$ value ($<0.05$) indicates that at least one predictor is associated with the outcome, without identifying which one.

Interpreting the Coefficients

Each coefficient $b_i$ is the change in $y$ for a one-unit increase in $x_i$, holding all other predictors in the model constant. This "holding constant" clause is what separates a multiple-regression coefficient from a simple correlation or a bivariate regression slope: it is an adjusted association, not a raw one, and the two can differ substantially — sometimes even in sign — when the predictors are themselves correlated with each other. MedCalc reports each coefficient with its standard error, 95% confidence interval, $t$-value, and $P$ value, exactly as in simple regression.

Two further correlation measures describe how much a predictor contributes once the others are accounted for:

• The partial correlation $r_{\text{partial}}$ is the correlation between $x_i$ and $y$ after the linear effects of the other predictors have been removed from both variables. It answers: among the variation in $y$ that the other predictors cannot explain, how strongly is $x_i$ associated with what remains?
• The semipartial correlation $r_{\text{semipartial}}$ removes the other predictors from $x_i$ only, not from $y$. Its square is the proportion of the total variance in $y$ uniquely attributable to $x_i$ — the share of $R^2$ that would be lost if $x_i$ were dropped from the model.

Both can differ strikingly from the zero-order (simple) correlation between $x_i$ and $y$ alone — a predictor that looks weak on its own can turn out to matter a great deal once the right variable is adjusted for, and vice versa. The worked example below shows this directly.

Multicollinearity

When two or more predictors are themselves highly correlated, the regression cannot separate their individual effects reliably: standard errors inflate, coefficients become unstable, and small changes in the data can swing a coefficient's sign or magnitude. This is multicollinearity. MedCalc quantifies it with the Variance Inflation Factor (VIF) for each predictor:

where $R_i^2$ is the $R^2$ obtained from regressing $x_i$ on all the other predictors in the model. A VIF near 1 indicates little collinearity; values above 5–10 are conventionally treated as cause for concern. When a predictor's tolerance ($1/\text{VIF}$) falls below 0.0001, MedCalc excludes it from the model altogether, because its coefficient can no longer be estimated reliably. Common remedies are to drop one of the correlated predictors, combine them into a single composite measure, or — if both are of genuine clinical interest — report the finding with the caveat that their individual contributions cannot be separated in this sample.

Confounding and Adjustment

A variable confounds the association between a predictor of interest and an outcome when it is related to both. Age is a classic example: older patients tend to have both higher LDL cholesterol and higher blood pressure, so a raw correlation between LDL and blood pressure will partly reflect their shared relationship with age rather than any direct connection between the two. Left unadjusted, a confounded association can be too large, too small, in the wrong direction, or entirely spurious.

Multiple regression addresses this directly: including the confounder as a predictor yields a coefficient for the variable of interest that is adjusted for the confounder — an estimate of the association that would be observed if the confounder did not vary across the sample. This is the same logic that produces the adjusted odds ratio in logistic regression, applied here to a continuous outcome. Adjustment can only account for confounders that were actually measured; unmeasured confounding cannot be corrected by any regression model, however many variables it includes.

Because adjustment is often the point of the analysis — not merely a nuisance to control for — the choice of which variables to adjust for should be made on scientific grounds, before the data are analysed: known or biologically plausible confounders belong in the model regardless of whether their own coefficient turns out to be statistically significant. This principle is directly at odds with the automated variable-selection procedures described next.

Why Stepwise Selection Is Not Recommended

MedCalc offers several ways to decide which predictors enter the model: Enter (all candidate variables are included in one step), Forward (variables are added one at a time as long as they are statistically significant), Backward (all variables start in the model and non-significant ones are removed), and Stepwise (a combination of the two, adding and removing variables as their significance changes at each step). These procedures are widely available in statistical software and widely used — but automated selection based on repeated significance testing is not a sound way to build a model, and should not be the default choice.

The problem is a version of the multiple testing problem described elsewhere in this book. Each step tests several candidate variables and keeps whichever clears the significance threshold; over the course of building the model, dozens of hypothesis tests may effectively be performed on the same dataset, but only the "winners" are reported, with no correction for how many comparisons produced them. The consequences are well documented (Harrell, 2015): reported $P$ values understate how likely the result is to be due to chance, standard errors are too small, $R^2$ is biased upward, and the coefficients of retained variables tend to be inflated in magnitude. Perhaps most damaging in practice, the selected set of variables is often unstable — refitting the same procedure on a bootstrap resample of the same data frequently selects a visibly different subset of "significant" predictors, which means the final model reflects the particular dataset as much as it reflects the underlying relationship.

Automated selection also cannot distinguish a variable that is a known or plausible confounder from one that is merely a candidate predictor: it treats every variable's fate as depending only on its $P$ value in this particular sample, discarding the prior knowledge that should guide adjustment (see Confounding and Adjustment above). A recognised confounder can be dropped by a stepwise procedure simply because, in this sample, its coefficient did not reach significance — leaving the remaining coefficients confounded despite the appearance of a rigorously selected model.

The recommended alternative is to specify the model's variables in advance, guided by the aims of the study and by prior evidence — not by their association with the outcome in the current dataset. A useful rule of thumb is to keep the number of predictors small relative to the sample size, on the order of at least 10 to 15 observations per predictor, to avoid an overfitted model with unstable coefficients. When the number of candidate predictors genuinely exceeds what the sample can support, shrinkage methods such as the LASSO — which select and estimate coefficients jointly, without relying on a sequence of significance tests — are generally preferable to stepwise procedures, though they are not implemented in MedCalc.

Residual Diagnostics

Multiple regression assumes that the residuals — the differences between observed and fitted values — are approximately normally distributed, with roughly constant variance across the range of fitted values. MedCalc can save the residuals as a new spreadsheet variable, which can then be examined with a box plot, a normal Q-Q plot, or a formal test for normality. As with any normality assessment, formal tests have limited power in small samples and excessive power in large ones; visual inspection of the residuals usually gives the more useful answer (see Interpreting the Tests). A funnel-shaped spread of residuals across the range of fitted values signals heteroscedasticity, for which MedCalc offers a weighted-regression option (Neter et al., 1996).

Worked Example

LDL cholesterol is measured in 12 patients, along with their age and body mass index (BMI):

Fitting LDL as a function of both age and BMI gives:

with $R^2 = 0.921$ and adjusted $R^2 = 0.903$: the two predictors together account for over 90% of the variation in LDL in this sample, even after the penalty for using two predictors instead of one. The residual standard deviation is 3.41 mg/dL. Both coefficients are highly significant:

Each additional year of age is associated with a 0.92 mg/dL increase in LDL, and each additional unit of BMI with a 2.03 mg/dL increase, after adjusting for the other.

The zero-order correlations tell a different story on their own: age correlates strongly with LDL ($r = 0.73$), but BMI's simple correlation with LDL looks weak ($r = 0.26$). Adjustment reveals why. Age and BMI are themselves negatively correlated in this sample ($r = -0.42$), so BMI's raw association with LDL is partly cancelled by its tendency to be lower in the older, higher-LDL patients. Once age is held constant, the partial correlation of BMI with LDL rises to 0.91 — the confounding relationship between age and BMI was masking BMI's true contribution. This is precisely the situation the Confounding and Adjustment section describes: a predictor's importance can be badly misjudged from its zero-order correlation alone.

Neither predictor shows a problematic Variance Inflation Factor (VIF = 1.21 for both — with exactly two predictors, they necessarily share a single value): the modest negative correlation between age and BMI here is not a threat to the stability of the coefficients.

MedCalc fits the model, and reports $R^2$ and adjusted $R^2$, the regression coefficients with partial and semipartial correlations, the Variance Inflation Factor, and diagnostic residual plots, in its Multiple regression procedure.

Literature

• Harrell FE (2015) Regression modeling strategies. 2nd ed. New York: Springer.
• Neter J, Kutner MH, Nachtsheim CJ, Wasserman W (1996) Applied linear statistical models. 4th ed. Boston: McGraw-Hill.