Outlier detection
| Command: | Statistics Outlier detection |
Description
Outlier detection is used to detect anomalous observations in sample data.
Required input
Variable: the name of the variable containing the data to be analyzed.
Filter: (optionally) a filter in order to include only a selected subgroup of cases in the statistical analysis.
Methods of outlier detection:
- Grubbs - left-sided: check only the smallest value (Grubbs, 1969).
- Grubbs - right-sided: check only the largest value (Grubbs, 1969).
- Grubbs - double-sided: check the most extreme value at either side (Grubbs, 1969).
The single-sided Grubbs' tests are more sensitive than the double-sided test.
- Generalized ESD test: the Generalized Extreme Studentized Deviate (ESD) procedure can detect multiple outliers in one step (Rosner, 1983).
- test for maximum number of outliers: enter the maximum number of outliers to detect.
- Tukey: check for multiple outliers at either side, categorized as 'outside' or 'far out' values (Tukey, 1977).
- An outside value is defined as a value that is smaller than the lower quartile minus 1.5 times the interquartile range, or larger than the upper quartile plus 1.5 times the interquartile range (the 'inner fences').
- A far out value is defined as a value that is smaller than the lower quartile minus 3 times the interquartile range, or larger than the upper quartile plus 3 times the interquartile range (the 'outer fences').
John Tukey, in fact, did not use the term 'outlier', but used the classifications 'outside' and 'far out'.
Options
- Alpha level for Grubbs' and ESD test: select the alpha-level (ranging from 0.10 to 0.001), applicable only in Grubbs' test and the Generalized ESD test. With a bigger alpha-level the test will be more sensitive and outliers will more rapidly be detected; however, this may result in false-positive results.
- Logarithmic transformation: the outlier detection methods assume that the data follow an approximately normal distribution (see next option). Sometimes data should be logarithmically transformed before analysis. See Logarithmic transformation. The example on this page uses the data from Rosner (1983) on their original scale. Therefore Logarithmic transformation is performed like in the Rosner paper.
- Test for Normal distribution: see Tests for Normal distribution.
Results
Outlier detectionVariable
Vit_E_Intake
Vit E IntakeBack-transformed after logarithmic transformation.
Sample size
54
Lowest value
0.7800
Highest value
407.4800
Geometric mean
10.1834
Median
8.1249
Coefficient of Skewness
1.1817 (P=0.0011)
Coefficient of Kurtosis
1.9972 (P=0.0248)
Shapiro-Francia test
for Normal distributionW'=0.9000
reject Normality (P=0.0006)Suspected outliers
Grubbs - double-sided (alpha-level 0.05)
None
Tukey, 1977
Outside values
208.51 225.88 407.48
Far-out values
None
Generalized ESD test (alpha-level 0.05)
208.51 225.88 407.48
Box-and-Whisker plot Summary statistics
- Summary statistics for the selected data are displayed. See Summary statistics.
- If the test for Normal distribution reports 'reject Normality' the outlier detection methods may be invalid since they assume that the data follow an approximately normal distribution. Perhaps data should have been logarithmically transformed before analysis. In the example, data are logarithmically transformed.
Suspected outliers
The program lists the outliers identified by the different procedures.
Grubbs' test can only be used to detect one single outlier; if you suspect there is more than one outlier you should not repeat the procedure but use the Generalized ESD test.
What to do when you have identified an outlier
Do not remove outliers automatically.
- Remove outliers only when a cause can be found for the spurious result, such as a pre-, post-, or analytical error. When you conclude that a pre-, post-, or analytical error is the cause of the spurious result, be aware that the same errors may exist in the other data values.
- Check the distribution of the data. Logarithmically transformed sample data may more closely follow a Normal distribution. Graph the data with and without logarithmic transformation, for example using a Box-and-Whisker plot.
- You may consider to replace the outlier value with the next highest/lowest (non-outlier) number.
- Keep the outlier but use robust or nonparametric statistical methods that do not assume that data are Normally distributed.
- Do the statistical analysis and report conclusions both with and without the suspected outlier.
In all cases, report the outliers and how you have dealt with them.
Literature
- Grubbs FE (1969) Procedures for detecting outlying observations in samples. Technometrics 11:1-21.
- Rosner B (1983) Percentage points for a generalized ESD many-outlier procedure. Technometrics 25:165-172.
- Tukey JW (1977) Exploratory data analysis. Reading, Mass: Addison-Wesley Publishing Company.
See also
Recommended book
Outliers in Statistical Data
Barnett V, Lewis TBuy from Amazon
From its initial publication this book has been the standard text on the subject. Since then there has been a continuing high level of activity, and work has developed in all major areas. This third edition reflects the latest state of knowledge with fully revised and extended coverage of all topics. Additional topics and new emphases are presented and a richer coverage of practical fields and computer-based facilities, together with a fully updated reference list, are provided.
