Outlier Detection
An outlier is an observation that lies an abnormal distance from other values in a random sample. Outliers can be caused by measurement errors, data entry mistakes, or genuine, rare anomalies in the underlying phenomenon being studied.
Deciding what to do with an outlier is one of the most consequential judgements in data analysis. If an outlier arises from a transcription error or a calibration fault, removing or correcting it is clearly justified. But if it is a genuine biological extreme — an unusually high troponin in a patient who truly had a severe infarction, for example — discarding it would distort the picture of reality. Blindly removing outliers without investigation is a form of data manipulation; retaining values that are almost certainly erroneous undermines the validity of any analysis. Formal outlier tests help make this decision transparent and reproducible rather than subjective.
Outliers matter because most classical statistical methods — means, standard deviations, Pearson correlations, linear regression — are sensitive to extreme values. A single erroneous data point can substantially shift the mean, inflate the standard deviation, and distort a regression line, leading to misleading conclusions from an otherwise sound study.
The IQR Rule (Tukey's Method)
The most common non-parametric method for identifying outliers using a box plot is the IQR rule. A data point $x$ is flagged as an outlier if it falls outside the following boundaries:
$$ \text{Lower Bound} = Q_1 - 1.5 \times IQR $$ $$ \text{Upper Bound} = Q_3 + 1.5 \times IQR $$Any data point smaller than the Lower Bound or larger than the Upper Bound is plotted individually as a dot on the box plot.
The choice of the multiplier 1.5 was made by the statistician John Tukey on empirical grounds. For data that follow a normal distribution, boundaries set at $Q_1 - 1.5 \times IQR$ and $Q_3 + 1.5 \times IQR$ extend to approximately ±2.7 standard deviations from the mean, meaning that only about 0.7% of genuinely normal observations would be flagged as outliers. This makes the rule conservative enough to avoid excessive false alarms while still catching values that are genuinely extreme. A stricter criterion uses a multiplier of 3.0, identifying only the most extreme observations as "far outliers".
The IQR rule is non-parametric: it makes no assumption about the underlying distribution. This makes it broadly applicable and easy to visualise on a box plot, but it is less powerful than formal statistical tests when the normality assumption is actually met.
Grubbs' Test
Grubbs' test (also called the extreme studentized deviate test) is a formal statistical test for detecting a outlier in a univariate dataset that follows an approximately normal distribution. Both Grubbs' test and the GESD test below assume normality; if the data are clearly non-normal, applying a transformation or a non-parametric approach is preferable. Methods for assessing normality are covered in the Are the data normally distributed? chapter. It tests whether the value most extreme from the mean is a statistically significant outlier.
The Grubbs test statistic $G$ is calculated as the largest absolute deviation from the sample mean, divided by the sample standard deviation:
$$ G = \frac{\max_{i} |x_i - \bar{x}|}{s} $$The null hypothesis is that there are no outliers. If $G$ exceeds the critical value for the chosen significance level $\alpha$ and sample size $n$, the most extreme value is declared an outlier. The critical value is derived from the $t$-distribution:
$$ G_{\text{crit}} = \frac{n-1}{\sqrt{n}} \sqrt{\frac{t_{\alpha/(2n),\, n-2}^{2}}{n - 2 + t_{\alpha/(2n),\, n-2}^{2}}} $$where $t_{\alpha/(2n),\, n-2}$ is the critical value of the $t$-distribution with $n-2$ degrees of freedom and a significance level of $\alpha/(2n)$ (two-tailed Bonferroni correction).
Example: A laboratory measures serum calcium in 20 patients. The result 15.8 mg/dL is suspected to be erroneous. Grubbs' test formally evaluates whether this single extreme value is a statistically significant outlier, helping decide whether to repeat the measurement before reporting.
Limitation: Grubbs' test is designed to detect . Running it repeatedly on the same dataset inflates the type I error rate, a problem known as masking.
Generalized ESD Test
The Generalized Extreme Studentized Deviate (GESD) test, proposed by Rosner (1983), extends Grubbs' test to detect up to $r$ outliers simultaneously in a normally distributed dataset, where $r$ is specified in advance. It avoids the masking problem that arises from applying Grubbs' test repeatedly.
The procedure performs $r$ iterations. At each step $i$ (from 1 to $r$), the test statistic is the same Grubbs-type ratio applied to the remaining data:
$$ R_i = \frac{\max_{j} |x_j - \bar{x}_i|}{s_i} $$where $\bar{x}_i$ and $s_i$ are the mean and standard deviation computed after removing the $i-1$ most extreme values found in previous steps. The corresponding critical value at step $i$ is:
$$ \lambda_i = \frac{(n - i)\, t_{p,\, n-i-1}}{\sqrt{(n - i - 1 + t_{p,\, n-i-1}^{2})(n - i + 1)}} $$where $p = 1 - \frac{\alpha}{2(n - i + 1)}$. After all $r$ statistics are computed, the number of outliers is determined by finding the largest $i$ for which $R_i > \lambda_i$; all $i$ values up to that point are declared outliers.
Example: In a multi-centre clinical trial measuring fasting glucose in 150 patients, the data-cleaning team suspects up to 5 values may be erroneous (e.g., unit confusion between mg/dL and mmol/L). Setting $r = 5$ and running GESD identifies all suspicious values in a single, multiplicity-corrected procedure.
MedCalc implements the IQR rule, Grubbs' test, and the Generalized ESD test in its Outliers procedure.