α-correction for simultaneous statistical inference:
Familywise error rate vs. per-family error rate
A meta-analysis of more than 30000 published articles indicated that less than 1% applied α-
corrections for multiple comparisons even though the median number of hypothesis tests per
article was ≈ 9 (Conover, 1973; Derrick & White, 2017; Pratt, 1959). A crucial, yet underap-
preciated difference, is the distinction between 1) the familywise (or experimentwise) error rate
(FWER), and 2) the per-family error rate (PFER). FWER is the probability of making at least
one Type I error in a family of hypotheses. The PFER, on the other hand, which is the number
α-errors expected to occur in a family of hypotheses (in other words, the sum of the probabili-
ties of α-errors for all the hypotheses in the family).The per-comparison error rate (PCER) is
the probability of a α-error in the absence of any correction for multiple comparisons
(Benjamini & Hochberg, 1995). Moreover, the false discovery rate (FDR) quantifies the ex-
pected proportion of "discoveries" (rejected null hypotheses) that are false (incorrect rejec-
tions).
The majority of investigations focus on the former while the latter is largely ignored even
though it evidently is at least equally important if not more so (Barnette & Mclean, 2005;
Kemp, 1975). The experimentwise (EW) error rate does not take the possibility of multiple α-
errors in the same experiment into account. Per-experiment (PE) α-control techniques control
α for all comparisons (a priori and post hoc) in a given experiment. In other terms, they con-
sider all possible α-errors that in a given experiment. It has been persuasively argued that per-
experiment α control is most relevant for pairwise hypothesis decision-making (Barnette &
Mclean, 2005) even though most textbooks (and researchers) focus on the experimentwise er-
ror rate. Both approaches differ significantly in the way they adjust α for multiple hypothesis
tests. It has been pointed out that the almost exclusive focus on experimentwise error rates is
not justifiable (Barnette & Mclean, 2005). From a pragmatic point of view, per-experiment
error correction is much closer aligned with prevailing research practices. In other words, in
most experiments it is not just the largest difference between conditions which is of empirical
interest and most of the time all pairwise comparisons are computed. The EW error rate treats
each experiment as one test even though multiple comparisons might have been conducted. A
systematic Monte Carlo based comparison between four different adjustment methods showed
that, for experimentwise control, Tukey’s HSD is the most accurate procedure (as an unpro-
tected test). If experimentwise α-control is desired, Tukey’s HSD (unprotected) test is the most
accurate procedure. If the focus is on per-experiment α-control, the Dunn-Bonferroni (again
unprotected) is the most accurate α-adjustment procedure (Barnette & Mclean, 2005).
