One-Way Analysis of Variance (ANOVA)

When the means of three or more independent groups are to be compared, the two-sample t-test cannot be applied directly: performing all pairwise t-tests simultaneously inflates the Type I error rate in proportion to the number of comparisons. One-way analysis of variance (ANOVA) addresses this by testing all groups in a single omnibus F-test — asking whether the variability between group means is larger than would be expected from the variability within groups alone. A significant result indicates that at least one group mean differs from the others, after which post-hoc tests identify which specific pairs are responsible. When the normality or equal-variance assumptions of ANOVA cannot be justified, the Kruskal–Wallis test provides a non-parametric alternative.

The Core Idea: Partitioning Variability

The fundamental insight of ANOVA is that the total variability in a dataset can be split into two components: variability $\mathrm{between}$ groups (caused by genuine differences in group means, plus random error) and variability $\mathrm{within}$ groups (caused by random variation among subjects within the same group). If the null hypothesis is true — all group means are equal — both components should reflect nothing more than random variation, and their ratio should be close to 1. If at least one group mean truly differs, the between-group component will be inflated relative to the within-group component, and the ratio will exceed 1.

This ratio is the F-statistic. It is compared to the F-distribution to obtain a P value. A small P value — conventionally below 0.05 — leads to rejection of the null hypothesis that all population means are equal.

The ANOVA Table

Results are customarily presented in an ANOVA table. Suppose there are $k$ groups with a total of $n$ observations combined. The table has the following structure:

Each mean square is a sum of squared deviations divided by its degrees of freedom — it is effectively a variance estimate. The F-statistic is:

Under the null hypothesis, $F$ follows an F-distribution with $k-1$ and $n-k$ degrees of freedom. The within-group mean square — sometimes called the residual mean square or error mean square — is used as a pooled estimate of the common within-group variance, and its square root is the residual standard deviation, a useful summary of the unexplained variability in the data.

Assumptions

One-way ANOVA rests on three assumptions:

1. Independence. The observations within and across groups are independent of one another. This is a design requirement, not something that can be checked from the data; it is satisfied by proper randomisation and the absence of repeated measurements or clustering.
2. Normality. Within each group, the observations are drawn from a normally distributed population. When group sizes are reasonably large (roughly 20 or more per group), the central limit theorem makes the F-test robust to moderate departures from normality. With small groups, normality of residuals should be assessed using the methods in the Are the data normally distributed? chapter.
3. Equal variances (homoscedasticity). The population variance is assumed to be the same in all groups. The ANOVA F-test is reasonably robust to moderate inequality of variances when group sizes are equal, but can be misleading when group sizes differ substantially and variances differ by more than a factor of roughly three. Levene's test can be used to check this assumption, though it is worth noting that it can be overly sensitive with large samples.

When the equal-variance assumption is seriously violated, Welch's one-way ANOVA — an extension of Welch's two-sample t-test to multiple groups — is a more reliable alternative that does not require equal variances.

Post-Hoc Comparisons

A significant omnibus F-test establishes only that the group means are not all equal. It does not indicate which groups differ from which. To answer that question, post-hoc comparisons (also called multiple-comparison procedures) are applied after a significant ANOVA result.

The Student–Newman–Keuls (SNK) test orders the group means from smallest to largest and then tests pairs in a stepwise fashion, starting with the range spanning all $k$ groups and working inward to adjacent pairs. Because the critical value it uses depends on how many means apart the two groups being compared are, the SNK test is more powerful than procedures that use the same critical value for all comparisons. This extra power comes at a cost: the SNK test does not strictly control the family-wise error rate when $k > 3$, making it somewhat more likely to produce false positives in large experiments.

The Tukey–Kramer test (an extension of Tukey's HSD to unequal group sizes) compares every pair of means simultaneously using a single critical value derived from the studentised range distribution. This guarantees that the family-wise error rate is held at 5% across all pairwise comparisons, regardless of the number of groups. The Tukey–Kramer test is the natural default when all pairwise comparisons are of interest and control of the overall error rate is a priority. It also yields simultaneous confidence intervals for every pairwise difference.

Scheffé's test is the most conservative of the three. Rather than restricting comparisons to pairs of individual means, it controls the family-wise error rate for $\mathrm{all\; possible\; contrasts}$ — any weighted combination of group means, not just pairwise differences. This generality makes it the appropriate choice when the scientific question involves planned comparisons of subsets of groups (for example, combining two treatment arms versus a control), but it pays for that flexibility with lower power for simple pairwise tests. When only pairwise comparisons are needed, Scheffé is unnecessarily stringent.

The choice of post-hoc method should be made before inspecting the data, guided by the scientific question. For exploratory all-pairs comparisons with equal group sizes, the SNK test offers good power. For a rigorous family-wise error guarantee on all pairwise comparisons — particularly with unequal group sizes — Tukey–Kramer is preferred. Scheffé is reserved for experiments where complex contrasts between groups of means are of interest.

Effect Size

Statistical significance does not imply practical importance. The standard effect-size measure for ANOVA is eta-squared ($\eta^2$), which is the proportion of total variability in the outcome attributable to the group factor:

By convention, $\eta^2 \approx 0.01$ is a small effect, $\approx 0.06$ a medium effect, and $\approx 0.14$ a large effect. Eta-squared tends to overestimate the population effect size in small samples; the bias-corrected version, omega-squared ($\omega^2$), is preferred when a less biased estimate is needed.

Worked Example

A clinical trial randomises patients with chronic low back pain to one of three treatments: physiotherapy alone, analgesic medication alone, or combined therapy. After 12 weeks, pain is scored on a 0–100 visual analogue scale (lower scores indicate less pain); the results for 30 patients (10 per group) are summarised below:

The overall (grand) mean is $(52.4 + 46.8 + 34.2)/3 = 44.5$. The ANOVA table is:

The F-statistic of 13.7 on 2 and 27 degrees of freedom is highly significant ($P < 0.001$), indicating that not all three group means are equal. The residual standard deviation is $\sqrt{89.5} \approx 9.5$ VAS points, and $\eta^2 = 2455.2\,/\,4873.8 \approx 0.50$, a large effect.

Post-hoc Tukey HSD comparisons reveal:

• Combined therapy vs physiotherapy: mean difference $-18.2$ points (95% CI $-26.5$ to $-9.9$), $P < 0.001$.
• Combined therapy vs medication: mean difference $-12.6$ points (95% CI $-20.9$ to $-4.3$), $P = 0.003$.
• Physiotherapy vs medication: mean difference $5.6$ points (95% CI $-2.7$ to $13.9$), $P = 0.24$.

Combined therapy reduces pain substantially more than either monotherapy; the two monotherapies do not differ significantly from each other. These differences are both statistically significant and large relative to the measurement scale, suggesting genuine clinical importance.

MedCalc computes one-way ANOVA with post-hoc comparisons in its One-way analysis of variance procedure.