Ubiquitous statistical fallacies
Protected versus unprotected pairwise comparisons
Christopher B. Germann (Ph.D., M.Sc., B.Sc. / Marie Curie Alumnus)
URL: https://christopher-germann.de
It is generally regarded as “best practice” to compute post hoc pairwise multiple compar-
isons only after a significant omnibus F-test. Many widely sold textbooks either explicitly
or implicitly advocate the utilisation of protected tests before post hoc comparisons are
conducted (i.a., Kennedy & Bush, 1985; Maxwell & Delaney, 2004). That is, a 2-stage
strategy is advocated and widely adopted by most researchers as evidenced in the liter-
ature. The 2-stage strategy makes post hoc pairwise comparisons conditional on a sta-
tistically significant omnibus F-test (hence the name protected test). However, this rec-
ommendation is not evidence based and there is no analytic or empirical evidence in sup-
port of this practice. To the contrary, it has been empirically demonstrated that this
strategy results in a significant inflation of α-error rates (Keselman, Games, & Rogan,
1979). Further empirical evidence against the 2-stage (protected) testing strategy is
based on a Monte Carlo analysis which explicitly compared protected versus unprotected
testing procedures. Independent of the error control method used (i.e., Dunn-Šidák,
Dunn-Bonferroni, Holm, Tukey’s HSD) unprotected tests performed significantly better
compared to protected tests (Barnette & Mclean, 2005). This simulation study clearly
demonstrated that using the F-test as a “protected gateway” for post hoc pairwise com-
parison is overly conservative. The simulation results clearly show that protected tests
should not be used. Independent of weather experimentwise or per-experiment α-control
is used, and no matter which α-error control technique is used (i.e., Dunn-Šidák, Dunn-
Bonferroni, Holm, Tukey’s HSD, etc.) unprotected tests generally outperformed their
protected counterparts.
Based on this evidence, it can be safely concluded that unpro-
tected testing procedures should be preferred over 2-stage protected procedures. The
conventional wisdom of conducting omnibus tests before post hoc comparisons are per-
formed does not stand the empirical/mathematical test. The authors of the previously cited
Monte Carlo simulation study conclude their paper with the following statement: “
Only when
one is willing to question our current practice can one be able to improve on it” (Barnette
& Mclean, 2005, p. 452).
A neglectable exception was only the Holm procedure in the case of per-experiment error control (but
not in the case of experimentwise error control). In this specific constellation, α of .10 was more accu-
rate as a protected test as compared to an unprotected test. This accuracy difference was lower when α
was .05 or .01.
Barnette, J. J., & Mclean, J. E. (2005). Type I Error Of Four Pairwise Mean
Comparison Procedures Conducted As Protected And Unprotected Tests. Journal
of Modern Applied Statistical Methods, 4(2), 446–459.
Kennedy, J. J., & Bush, A. J. (1985). An introduction to the design and analysis of
experiments in behavioral research. Lanham, MD: University Press of America,
Keselman, H. J., Games, P. A., & Rogan, J. C. (1979). Protecting the overall rate of
Type I errors for pairwise comparisons with an omnibus test statistic.
Psychological Bulletin, 86(4), 884–888. https://doi.org/10.1037/0033-
Maxwell, S., & Delaney, H. (2004). Designing experiments and analyzing data: A
model comparison perspective. Briefings in functional genomics proteomics (Vol.
4). https://doi.org/10.1002/sim.4780100917