Effect Size Measures
The Clinical versus Statistical Significance chapter makes the case that a $P$-value alone cannot tell you whether an effect matters — only whether it is unlikely to be zero. Effect size is the answer to the question that actually matters: how large is the effect? This chapter collects the standardised effect-size measures used throughout this book — the standardised mean difference for comparisons of means, eta-squared for ANOVA, and Cramér's V and the phi coefficient for categorical association — and shows how each is calculated and interpreted. Odds ratios, relative risk, hazard ratios and the correlation coefficient $r$ are also effect sizes in the broad sense; they are covered in their own chapters and referenced here for completeness.
What unifies these measures is standardisation: each rescales a raw difference or association onto a unit-free scale, so that the magnitude of an effect can be judged on its own terms and compared across studies, outcomes, and measurement scales — something a raw mean difference in mmHg or a $\chi^2$ statistic cannot do, because both depend on the units of measurement or the sample size.
Standardised Mean Difference (Cohen's d)
The standardised mean difference (SMD) expresses the difference between two means in units of standard deviation rather than in the original units of measurement. This makes it possible to compare the size of an effect across studies that used different instruments or scales — a pain score in points and a pain score in millimetres on a visual analogue scale become directly comparable once expressed as an SMD.
Cohen's d for two independent groups
For two independent groups with means $\bar{x}_1$, $\bar{x}_2$ and standard deviations $s_1$, $s_2$, Cohen's $d$ is the mean difference divided by a standard deviation common to both groups:
$$ d = \frac{\bar{x}_2 - \bar{x}_1}{s_p}, \qquad s_p = \sqrt{\frac{(n_1-1)s_1^2 + (n_2-1)s_2^2}{n_1+n_2-2}} $$where $s_p$ is the same pooled standard deviation used in the two-sample $t$-test (see the Comparison of two groups of independent observations chapter). When the two group variances are noticeably unequal, pooling them is questionable, and the average of the two standard deviations, $s_{av} = \sqrt{(s_1^2+s_2^2)/2}$, is used in the denominator instead — this variant is denoted $d_{av}$.
Cohen's dz for paired observations
For paired data (see the Two groups of paired observations chapter), the SMD is calculated differently: the mean of the paired differences is divided by the standard deviation of the differences, not by a between-subject standard deviation:
$$ d_z = \frac{\bar{x}_{d}}{s_{d}} $$Because $s_d$ already reflects the correlation between the two paired measurements — it is smaller when the pairing is effective at removing between-subject variability — $d_z$ is not directly comparable to the $d$ computed for independent groups from the same raw mean difference and standard deviations. A paired design with strongly correlated measurements will show a substantially larger $d_z$ than the $d$ obtained by (incorrectly) treating the same data as independent. In the antihypertensive trial from the paired observations chapter, $\bar{d} = 11.75$ mmHg and $s_d = 4.54$ mmHg, so $d_z = 11.75 / 4.54 \approx 2.59$ — a very large effect, reflecting both a substantial mean reduction and the low pair-to-pair variability in the response to treatment.
Interpreting the magnitude
Cohen's (1988) original benchmarks remain the standard rough guide, applicable to $d$, $d_{av}$ and $d_z$ alike:
| $|d|$ | Interpretation |
|---|---|
| ≈ 0.2 | Small effect |
| ≈ 0.5 | Medium effect |
| ≥ 0.8 | Large effect |
These thresholds are a starting point for fields with no established convention, not a substitute for a domain-specific judgement of what constitutes a meaningful difference — the same role the minimal important difference plays for raw, unstandardised outcomes.
A confidence interval for $d$ should always accompany the point estimate. Because $d$ is a ratio of two estimated quantities, its sampling distribution is not simply normal; the standard approach uses the noncentral $t$-distribution (Goulet-Pelletier & Cousineau, 2018, for independent groups; Cousineau & Goulet-Pelletier, 2021, for paired designs), which is markedly more accurate than older closed-form approximations, particularly in small samples. A related bias-corrected variant, Hedges' $g$, multiplies $d$ by a small-sample correction factor that removes the slight upward bias of $d$ when $n$ is small; the two measures converge as sample size increases.
Worked example
Reusing the trial from the independent observations chapter: 15 patients per arm are randomised to two oral hypoglycaemic agents, with HbA1c reduction (percentage points) of mean 1.82 (SD 0.61) with Drug A and mean 1.18 (SD 0.84) with Drug B.
$$ s_p = \sqrt{\frac{14(0.61)^2 + 14(0.84)^2}{28}} = \sqrt{\frac{5.21 + 9.88}{28}} = \sqrt{0.539} = 0.734 $$ $$ d = \frac{1.82 - 1.18}{0.734} = \frac{0.64}{0.734} \approx 0.87 $$An SMD of 0.87 is a large effect by Cohen's benchmark. This is a useful complement to the earlier conclusion from the raw mean difference: the trial found a statistically significant difference (95% CI 0.09 to 1.19 percentage points, $P = 0.025$) whose lower bound is only marginally clinically important in absolute terms, yet which corresponds to a large effect when expressed relative to the variability of the outcome. The two perspectives — absolute difference against the minimal important difference, and standardised difference against Cohen's benchmarks — are complementary, not competing; reporting both, with confidence intervals, gives the fullest picture.
Eta-Squared
For comparisons of three or more groups, the SMD does not generalise directly — there is no single "difference" to standardise. The One-way ANOVA chapter introduces eta-squared ($\eta^2$) as the effect-size measure for this setting: the proportion of total variability in the outcome attributable to the grouping factor,
$$ \eta^2 = \frac{\text{SS}_{\text{between}}}{\text{SS}_{\text{total}}} $$with the same Cohen (1988) benchmarks of 0.01, 0.06 and 0.14 for small, medium and large effects. As with $d$, a confidence interval for $\eta^2$ should be reported alongside the point estimate; it is obtained from the noncentral $F$-distribution (Steiger, 2004). Because $\eta^2$ is a proportion of variance rather than a standardised difference, it is not on the same numeric scale as $d$ — the two should not be compared directly, though $\eta^2$ for a two-group comparison and $d$ for the same data are mathematically related and will agree in their qualitative conclusion.
For the worked example in the ANOVA chapter — three treatments for chronic low back pain, $\text{SS}_{\text{between}} = 2455.2$, $\text{SS}_{\text{total}} = 4873.8$:
$$ \eta^2 = \frac{2455.2}{4873.8} \approx 0.50 $$a large effect: half of the total variability in pain scores is attributable to which treatment a patient received.
Cramér's V and the Phi Coefficient
For association between two categorical variables, tested with the chi-squared test, the $\chi^2$ statistic itself is not an effect size — like the $F$-statistic and $t$-statistic, it grows with sample size even when the underlying association is weak, so it cannot be compared across tables of different sizes. Cramér's V rescales $\chi^2$ to a bounded, sample-size-independent measure of association strength:
$$ V = \sqrt{\frac{\chi^2}{n \times \min(r-1,\, c-1)}} $$where $n$ is the total sample size and $r$, $c$ are the number of rows and columns in the contingency table. $V$ ranges from 0 (no association) to 1 (perfect association — each row category is associated with exactly one column category). For a $2 \times 2$ table, $\min(r-1, c-1) = 1$ and the formula reduces to the phi coefficient, $\phi = \sqrt{\chi^2/n}$, which is numerically identical to Cramér's V in that special case.
As a rough guide:
| Cramér's V | Interpretation |
|---|---|
| ≤ 0.2 | Weak association |
| 0.2 – 0.6 | Moderate association |
| > 0.6 | Strong association |
A confidence interval for Cramér's V, obtained from the noncentral $\chi^2$ distribution (Smithson, 2003), should be reported alongside the point estimate for the same reason as for $d$ and $\eta^2$: with small or moderate samples, the sampling variability of V can be considerable, and the point estimate alone overstates the precision of the conclusion.
Worked example
Reusing the antibiotic trial from the chi-squared test chapter — a $2 \times 2$ table of antibiotic (A vs B) by outcome (cured vs not cured) in 220 patients, with $\chi^2 = 4.49$:
$$ V = \sqrt{\frac{4.49}{220 \times 1}} = \sqrt{0.0204} \approx 0.14 $$Despite a statistically significant result ($P = 0.034$), the association between antibiotic choice and cure is weak by the conventional guide. This is the categorical-data analogue of the large-trial example in the Clinical versus Statistical Significance chapter: statistical significance establishes that the association is unlikely to be due to chance, while Cramér's V shows that the practical strength of that association, on this data, is modest.
Other Effect Size Measures in This Book
Several effect-size measures native to specific study designs are covered in their own chapters rather than repeated here:
| Measure | Used for | Chapter |
|---|---|---|
| Pearson's $r$, Spearman's $r_s$ | Strength of linear or monotonic association between two continuous variables | Correlation |
| $R^2$ | Proportion of variance in the outcome explained by a regression model | Regression |
| Relative risk, odds ratio, risk difference, NNT | Magnitude of association between an exposure and a binary outcome | Risk, odds, effect measures and NNT |
| Hazard ratio | Relative rate of an event between groups over time | Cox proportional-hazards regression |
| Youden's $J$, AUC | Discriminative performance of a diagnostic test | ROC curve analysis |
Choosing and Reporting an Effect Size
The choice of effect-size measure follows directly from the design and the test already being used: the SMD accompanies a $t$-test, eta-squared accompanies ANOVA, Cramér's V or phi accompanies a chi-squared test. There is rarely a genuine choice to be made — the measure is dictated by the data structure, not selected independently of it.
What is a genuine choice is whether to report an effect size at all, and the answer should always be yes, together with its confidence interval. A $P$-value establishes that an effect is unlikely to be exactly zero; the effect size and its interval establish how large it plausibly is. Reporting one without the other — a significant $\chi^2$ with no Cramér's V, an ANOVA table with no $\eta^2$ — leaves the reader unable to judge whether a statistically confirmed effect is one worth acting on.
MedCalc reports the standardised mean difference (Cohen's $d$) with its confidence interval as an option in the Independent samples t-test and Paired samples t-test procedures, eta-squared with its confidence interval in the One-way analysis of variance procedure, and Cramér's V with its confidence interval in the Chi-squared test and Frequency table & Chi-squared test procedures.
Literature
- Cohen J (1988) Statistical power analysis for the behavioral sciences, 2nd ed. Hillsdale, NJ: Lawrence Erlbaum Associates.
- Cousineau D, Goulet-Pelletier J-C (2021) A study of confidence intervals for Cohen's dp in within-subject designs with new proposals. The Quantitative Methods for Psychology 17(1):51–75.
- Goulet-Pelletier J-C, Cousineau D (2018) A review of effect sizes and their confidence intervals, Part I: The Cohen's d family. The Quantitative Methods for Psychology 14(4):242–265.
- Smithson M (2003) Confidence intervals. Thousand Oaks, CA: Sage Publications.
- Steiger JH (2004) Beyond the F test: Effect size confidence intervals and tests of close fit in the analysis of variance and contrast analysis. Psychological Methods 9:164–182.