ThePitfallsofHypothesis Testing

Christopher B. Germann

Null Hypothesis Significance Tes ting

ALong‐St anding Issue

Bakan,D.,1966.Thetestofsignificanceinpsychologicalresearch.PsychologicalBulletin66,423–437.

Cohen,J.(1994).Theearthisround(p<.05).AmericanPsychologist,49,997‐1003.

Chow,S.L.,1998.Précisof“Statisticalsignificance:rationale,validity,andutility”.BehavioralandBrainSciences,21,169–239.

Falk,R.,Greenbaum,C.W.,1995.Significancetestsdiehard.TheoryandPsycholog

y5,75–98.

Lecoutre,M.P.,Poitevineau,J.,Lecoutre,B.,2003.EvenstatisticiansarenotimmunetomisinterpretationsofNullHypothesis

SignificanceTests.InternationalJournalofPsychology38,37–45.

Loftus,G.R.,1991.Onthetyrannyofhypothesistestinginthesocialsciences.ContemporaryPsychology36,102–105.

Nickerson,R.S.,2000.Nullhypothesissignificancetesting:areviewofanoldandcontinuingcontroversy.PsychologicalMethods

5,241–301.

Meehl,P.E.(1967).Theory

testinginpsychologyandphysics:Amethodologicalparadox.PhilosophyofScience,34,103‐115.

Morrison.D.E.,&Henkel,R.E.(Eds.).(1970).ThesignificancetestcontroversyChicago:Aldine.

Rozeboom,W.W.(1960).Thefallacyofthenullhypothesissignificancetest.Psycholog

icalBulletin,57,416‐428.

Stevens,S.S.,1960.Thepredicamentindesignandsignificance.ContemporaryPsychology9,273–276.

Waller,N.G.,2004.Thefallacyofthenullhypothesisinsoftpsychology.AppliedandPreventivePsychology11,83–86.

Experiment

Bayes

Fisher

Neyman &

Pearson

Fishervs.

Neyman &

Pearson

Hybrid

Theory

Publication

bias

Solutions

Outline

Onlineexperiment

Pleasevisitthefollo wingURL:

http://irrational‐decisions.com/?page_id=159

UsernameandPasswordarebothcognovo

Supposeyouhaveatreatmentthatyoususpectmayalterperformanceonacertain

task.

Youcomparethemeansofyourcontrolandexperimentalgroups(say20subjectsin

eachsample).Further,supposeyouuseasimpleindependentmeanst‐testand

yourresultissignificant(t=2.7,d.f. =18,p=0.01).Pleasemarkeachofthe

following

statementsas“true” or“false.”“False”meansthatthestatementdoes

notfollowlogicallyfromtheabovepremises.Alsonotethatseveralornoneofthe

statementsmaybecorrect.

Asimplet‐tests

Experiment

Bayes

Fisher

Neyman &

Pearson

Fishervs.

Neyman &

Pearson

Hybrid

Theory

Publication

bias

Solutions

Outline

Referend ThomasBayes

1701‐1761

BayestheoremwaspublishedpostmortembyhisfriendRichardPricein1763.

( ) ( )()

A B PABPB

ConditionalProbability

Pr (Ace&Blackcard)=Pr(Ace|Black)xPr(Black)

( ) ( | )()

BA PBAPA





ConditionalProbability

Pr (Blackcard&Ace)=Pr(Black|Ace)xPr(Ace)

NOTE:

(|) (|)PAB PB A



( ) ( )()

A B PABPB

()()()

BA PBAPA

( )() ( )() PABPB PBAPA





BAYESRULE

()()()/()

AB PBAP A PB

priorposterior

likelihood

marginalevidence

()()()/()

AB PBAPA PB

()()()/()

He PeHPH Pe

A→H=hypothesis

B→e=experimentalevidence

THEINVERSEPROBLEMOFPROBABILITY

211

4132











Prior:Fulldeck!



211

4102









Prior:Nopicturecards

()()()/()PHe PeHPH Pe

Yourestimationofthestateoftheworld

dependsonyourpriors(orpriorbeliefs)

()()

()

PeH PH

PH e



()()

()

PeH PH

PH e



()()

()()()()

PeH PH

PeH PH PeH PH





()()

()()()()

PeH PH

PeH PH PeH PH





2–AlternativeHypotheses

Example

1%ofwomenover40getbreastcancer

80%ofwomenwithbreastcancerhaveapositivemammography

10%ofwomenwithoutbreastcanceralsohaveapositivemammography

Awomantestspositive,whatistheprobabilityshehasbreastcancer?

()()()/()PH e PeH PH Pe

(e | ) 0.8PH ( ) 0.01PH



( ) ( ) ( ) ( ) ( ) (0.1)(0.99) (0.8)(0.01) 0.11Pe PeH PH PeH PH

111

000

()()()/()(0.8)(0.01)/0.110.08

()()()/()0.92

PH e PeH PH Pe





Bay e sianLiteracy

100womanov er40100womanov er40

1

breast

cancer

1

breast

cancer

0.8positive0.8positive 0.2negative0.2negative

99

nobreast

cancer

99

nobreast

cancer

9.9

positive

9.9

positive

89.1negative89.1negative

80%

20%

10%

90%

1% 99%

Manyphysiciancommitthesocalled“baseratefallacy”(neglectingtheaprori probability).

See,forexample,Bramwell R,WestH,SalmonP. (2006). Healthprofessionals’andusers’interpretationof

screeningtestresults:anexperimentalstudy.BritishMedicalJournal.333,284‐286.

BAYES’THEOREM

Thetheorythatwouldnotdie

Bayes'theoremcrackedtheenigmacode

(themodernBaysian revival)

HunteddownRussiansubmarines

Emergedtriumphantfromtwocenturies

ofcontroversy!Itisnowheavilyusedin

manyfields...

Experiment

Bayes

Fisher

Neyman &

Pearson

Fishervs.

Neyman &

Pearson

Hybrid

Theory

Publication

bias

Solutions

Outline

Letmein troduce...

SirRonaldFisher 

(1890‐ 1962)

HewasaneminentEnglishgeneticist,

statistician,andevolutionarybiologist.

•Professorof Eugenics atthe University

CollegeLondon

•laterChairofGeneticsattheUniversityof

Cambridge

First published 1935First published 1925

Fisher’sseminalbook s

Fishervs.Bay es

FisherexplicitlyrejectedtheBayesianapproachbuttacitlyuses

Bayesianlogicinhisownreasoning

Fisher'squasi‐Bay e sianvie w

Althoughheisoftendescribedasafrequentist,whateverphraseology

heused,healwaysheldthatasignificantresultaff ectsourconfidence

ordegreeofbelief thatthenullhypothesisisfalse.

Fisherian nullhypothesis

t esting

Setupastatisticalnullhypothesis.

Thenullneednotbeanil

hypothesis(i.e.,zerodifference)

andtrytorejectit.

Theladytasting tea

Why5%signific ancelev e l?

InTheDesignFisherwrites:

“Itisusualandconvenientforexperimenterstotak e5%asastandard

levelofsignificance,inthesensethattheyarepreparedtoignoreall

resultswhichfailtoreachthisstandard...”(Fisher,1935,p.13)

“...noscientificwork erhasa fixedlevelofsignificanceatwhichfrom

yea

rtoyear,andinallcircumstances,herejectshypotheses;herather

giveshismindtoeachparticularcaseinthelightofhisevidenceandhis

ideas.”(Fisher,1956,p.41)

However,FisherwasnottheonlyonecontributingtothedevelopmentofNHST...

Experiment

Bayes

Fisher

Neyman &

Pearson

Fishervs.

Neyman &

Pearson

Hybrid

Theory

Publication

bias

Solutions

Outline

Ne yman&Pearson

JerzyNeyman(1894‐ 1981)

EgonPearson(1895‐ 1980)

VirtuallyallstaticstextbooksdonotmentionNeyman andPearsonin

thecontextofNHST!

Neyman‐Pearsondecision

theory

State of the World

true H

false

Research

Decision

Reject H

Type I error (α)

Correct

RRejection (1-

β)

(Power)

Accept H

Correct

Acceptance

Type II error (β)

The4Outc omes

Type1Err or

/2

1‐

Pr(H1|W0):theprobabilitythatweconcludeH1istruegiventhat

theworldisinstateW0=

TYPE1error=



Probabilityofrejectingthenullhypothesisincorrectly

Power&Type2Error

H0 H1







1‐



=POWER

Effectsize

Effect Size





H0 H1







1‐



=POWER

PowerDependsontheEffectSize

Effectsize=1s.d.

H0 H1







1‐



=POWER

PowerDependsontheEffectSize

Effectsize=3s.d.

H0 H1



PowerDependsonAlpha

Effectsize=0.5s.d.



1‐



=POWER

H0 H1



PowerDependsonAlpha

Effectsize=0.5s.d.



1‐



=POWER

Neyman–PearsonDecisionRule

(Rej| )PH





AswithFisher,N‐Pintroducedtheideaofadecisionrule:torejectthenull

hypothesisinfavourofthealternativehypothesis,Rej,ortoNOTrejectthe

nullhypothesisin favourofthenullhypothesis.

Theydidthisinaparticular way(nottheonlypossibility)

(Rej )PH

set

Thenmaximise

Neyman–PearsonDecision

Rule

1.Setuptwostatisticalhypotheses,H

andH

,anddecideaboutα,β,and

samplesizebeforetheexperiment,basedonsubjectivecost‐benefit

considerations.Thesedefinearejectionregionforeachhypothesis.

2.IfthedatafallsintotherejectionregionofH

,acceptH

;otherwise

acceptH

.Notethatacceptingahypothesisdoesnotmeanthatyou

“believe”init,butonlythatyouact“asifitweretrue”.

3.Theusefulnessoftheprocedureislimitedamongotherstosituations

whereyouhaveadisjunctionofhypotheses(e.g.,eitherμ

=8orμ

=10is

true)andwhereyoucanmakemeaningfulcost‐benefittrade‐offsfor

choosingalphaandbeta.

4.NotBayesian(nopriors),(butcanberelatedtoposteriorprobability).

Experiment

Bayes

Fisher

Neyman &

Pearson

Fishervs.

Neyman &

Pearson

Hybrid

Theory

Publication

bias

Solutions

Outline

Fishervs.Neyman&Pearson

Bothcampsinthecontroversyaccusedtheother

partyofmechanical,thoughtlessstatistical

inference.

Asymmetricvs.symmetric

hypothesestesting

Fisherspecifiedonlyonehypothesis,thenull,andleftunspecifiedthe

alternativehypothesis,typicallythehypothesistheresearcherisinterested

in.Thismadenonsignificance appearanegative,worthless,and

disappointingresult.

InNeyman‐Pearsontheory,bycontrast,thereissymmetry,andaconclusion

isdrawnfromnonsignificance.

Hiddenc onflicts

Theseconflictingviewsarealmostunknowntopsychologists.Textbooks

areuniformlysilent.

Theydon’tspelloutdifferencesbetweenFisher,Neyman andPearson

butinsteadpresentahybridversionwhichbothpartieswouldnothave

agreedupon.

Experiment

Bayes

Fisher

Neyman &

Pearson

Fishervs.

Neyman &

Pearson

Hybrid

Theory

Publication

bias

Solutions

Outline

TheoffspringofFisherandNP:

Thecurr en tlyusedhybridtheory

“AmishmashofFisherandNeyman‐Pearson,withinvalid

Bayesianinterpretation”(Cohen,1994,p.998)

Cohen,J.(1994).Theearthisround(p<.05).AmericanPsychologist,49,997‐1003.

1. Setuponly1hypothesis(H

)(inaccordwithFisher).

2. Use5%asaconventionforrejectingthenull.Ifsignificant,accept

yourresearchhypothesis.Reporttheresultasp<0.05,p<0.01,or

p<0.001(whichevercomesnexttotheobtainedp‐value).

3. Makeayes/nodecision(congruentwithN‐Pbutintheirtheorythe

signific

ancelev

elisnotfixedbyconventionbutbythinkingaboutα,

β,andthesamplesize).

4. Alwaysperformthisprocedure

TheHybrid

Criticism

FormerAPA presidentPaulMeehl (1978)putit

strongly:

“IsuggesttoyouthatSirRonaldhasbefuddledus,

mesmerizedus,andledusdowntheprimrosepath.I

believethatthealmostuniversalrelianceonmerely

refutingthenullhypothesisisoneoftheworstthings

thateverhappenedinthehistoryof p

sychology.”(p.

817)

JacobCohenarguesthatnullhypothesissignificance

testing„notonlyfailstosupporttheadvanceof

psychologyasasciencebutalsohasseriouslyimpeded

it.“(Cohen,1997,p.997)

GerdGigerenzer

“Fewresearchersareawarethattheirownheroesrejectedwhatthey

practiceroutinely.Awarenessoftheoriginsoftheritualandofits

rejectioncouldcauseavirulentcognitivedissonance,inadditionto

dissonancewitheditors,reviewers,anddearcolleagues.Suppressionof

conflictsandcontradictinginformationisintheverynatureofthissocial

ritual.”(Giger

enzer,2004,p.592)

In t erimsummary:

Wha tiswrongwithNHST?

Itdoesnottelluswhatwewanttoknow!Whatwewanttoknowis

"Giventhesedata,whatistheprobabilitythatH

istrue?"

Whatittellsusis"GiventhatH

istrue,whatistheprobabilityofthese

(ormoreextreme)data?“

Thesetwostatementsarenotthesameashasbeenpointedoutmany

timesovertheyearsbyMeehl (1978,1986,1990a,1990b),Gigerenzer

(1993),andCohen(1990).

Analy sis anddiscussionof

experimen talr esults

t=2.7,d.f. =18,p=0.01

1.Youhaveabsolutelydisprovedthenullhypothesis(thatis,thereisnodifference

betweenthepopulationmeans).

2.Youhavefoundtheprobabilityofthenullhypothesisbeingtrue.

3.Youhaveabsolutelyprovedyourexperimentalhypothesis(thatthereisadifference

betweenthepopulationmeans).

4.Youcandeducetheprobabilityoftheexpe

rimentalhypothesisbeingtrue.

5.Youknow,ifyoudecidetorejectthenullhypothesis,theprobabilitythatyouare

makingthewrongdecision.

6.Youhaveareliableexperimentalfindinginthesensethatif,hypothetically,the

experimentwererepeatedagreatnumberoftimes,youwouldobtainasignificantr

esult

on99%ofoccasions.

Oakes,M.(1986).Statisticalinference:Acommentaryforthesocialandthebehavioral

sciences.Chichester,England:Wiley.

Logic alfallacies

Recallthatap‐valueistheprobabilityoftheobserveddata(orofmore

extremedatapoints),giventhatthenullhypothesisH

istrue,defined

insymbolsasp(D|H

).

Statements1and3areeasilydetectedasformalfallacies,becausea

significancetestcanneverdisprovethenullhypothesisorthe

(undefined)experimentalhypothesis.1and3areinstancesofthe

illusionofcertainty(Gigerenzer,2002).

Logic alfallacies

Statements2and4arealsofalse.Theprobabilityp(D|H

)isnotthe

sameasp(H

|D),andmoregenerally,asignificancetestdoesnot

provideaprobabilityforahypothesis.

Statement5alsoreferstoaprobabilityofahypothesis.Thisisbecause

ifonerejectsthenullhypothesis,theonlypossibilityofmakingawrong

decisionisifthenullhypothesisistrue.Thus,itmakesessentia

llythe

sameclaimasStatement2does,andbothareincorrect.

Statement6amountstothereplicationfallacy(Gigerenzer,1993,2000).

Here,p=1%istak entoimplythatsuchsignificantdatawouldreappear

in99%oftherepetitions.

Replic a tionfallacy

thebeliefthatthelevelofsignificancegivesinformationaboutthe

replicabilityofanexperiment.

P(D|H

)doesnotimplyanything aboutp(Replication)

TheeditoroftheJournalofExperimentalPsychology,ArthurMelton,

statedthatheusedthelevelofsignificancereportedinsubmitted

papersasthemeasureofthe“confidencethattheresultsofthe

experimentwouldberepeatableundertheconditionsdescribed”

(Melton,1962,p.553).

Or,“Ifthe

statisticalsignificanceisatthe0.05level...theinvestigator

canbeconfidentwithoddsof95outof100thattheobserved

differencewillholdupinfutureinvestigations”(Nunnally,1975,p.195).

Perpe tua tingstatistical

illusions

Guilford,J.P.,1942.Fundamental

StatisticsinPsychologyandEducation,

3rded.,1956;6thed.,1978(with

Fruchter,B).McGraw‐Hill,NewYork.

InGuildfords handspvaluesturn

miraculouslyintoBayesianposterior

probabilities.

“Iftheresultcomesoutoneway,the

hypothesisisprobablycorrect,ifit

comesoutanotherway,thehypothesis

isprobablywrong”(p.156).

80%ofthestatisticsteachers

sharedillusionswiththeir

s tuden ts!

Thepermanen tillusion–

Inv erseprobability

p(D|H

)≠p(H

|D)

Nullhypothesissignificancetestingcanonlytellusp(D|H

),thatis,the

probabilityofthedatagiventhenullhypothesis.

Wecannotderivetheinverse(posteriorBayesian)probabilityp(H

|D),

theprobabilityofahypothesisgiventhedata!

Wecannotevencalculatep(D|H

),aswecaninNeyman andPearson’ s

theorybecauseH

hasnotbeenspecified.

WishfulBay esianthinking

... isthebeliefthatthelevelofsignificance,say.05,istheprobability

thatthenullhypothesisiscorrect.

Orconsequentlythat1‐.05istheprobabilitythatthealternative

hypothesisiscorrect.

Necessaryillusions

Wearguethatsuchillusionsarenecessarytomaintainthedreamof

mechanizedinductiveinference.

Withoutillusions,wewouldseeclearlythatthehybridsimplygivesus

p(D|H

)andnothingmore!

Withoutillusions,theritualwouldbeeasilyrecognizedforwhatitis.

Experiment

Bayes

Fisher

Neyman &

Pearson

Fishervs.

Neyman &

Pearson

Hybrid

Theory

Publication

bias

Solutions

Outline

Aparable

concerning

editorial policies

Notknowing what is f alse and what is not the

r esear cher sees 125h ypotheses as true,45of which

ar e not.Theneg ativ eresults ar e much mor e r eliable

butless likely to be published.

Type1err or infla tion:Imagine a

tests on1000hypotheses 100 of

which are true

Thetests have af alse positive ra teof 5%.Tha t means

they pr oduce 45f alse positiv es(5%of 900).They

ha v eapow erof.8,sotheycanconfirmonly80ofthe

trueh ypotheses,pr oducing20neg a tiv es.

Notknowing what is f alse and what is not the

r esear cher sees 125h ypotheses as true,45of which

ar e not.Theneg ativ eresults ar e much mor e r eliable

butless likely to be published.

Experiment

Bayes

Fisher

Neyman &

Pearson

Fishervs.

Neyman &

Pearson

Hybrid

Theory

Publication

bias

Solutions

Outline

Thelogicofscien tificInfer ence

Thelessontobelearnedisthatnosinglesolutiontotheprocessof

inductiveinferenceexists.

AmagicalternativetoNHST,someothermechanicalritualtoreplaceit

doesn’texist.(Cohen,1997)

OtherApproaches

• Confidenceintervals

• MaximumLikelihood

• Bayesianstatistics

• Avoidmakingpiecewisedecisions

NHST

(inconsistenthybrid)

Visualisingdat a

Blumenthal90

Study

‐2 ‐1 012

Yukna98

Masters96

Borghetti93

Meadows93

Altiere 79

Meandifferencebetweentest&controlgroups

ForestPlots

BayesFactors

()

() ()()

PeH

PH e

PH e PeH PH









Posterior

oddsratio

BAYES

FACTOR

Priors

oddsratio

Conclusion

Statistical reasoningisanart andsodemandsbothmathematical

knowledgeandinformedjudgment.Whenitismechanized,aswiththe

institutionalizedhybridlogic,itbecomesritual,notreasoning.

Thankyouforyourattention!

An yquest ionsar ewelc ome...