Cluster based computational methods for mass univariate analyses of event related brain potentials/fields : a simulation study

(1)

http://wrap.warwick.ac.uk/

Original citation:

Pernet, C. R., Latinus, M., Nichols, Thomas E. and Rousselet, G. A.. (2014)

Cluster-based computational methods for mass univariate analyses of event-related brain

potentials/fields : a simulation study. Journal of Neuroscience Methods .

Permanent WRAP url:

http://wrap.warwick.ac.uk/65179

Copyright and reuse:

The Warwick Research Archive Portal (WRAP) makes this work of researchers of the

University of Warwick available open access under the following conditions.

This article is made available under the Creative Commons Attribution- 3.0 Unported

(CC BY 3.0) license and may be reused according to the conditions of the license. For

more details see

http://creativecommons.org/licenses/by/3.0/

A note on versions:

The version presented in WRAP is the published version, or, version of record, and may

be cited as it appears here.

(2)

ContentslistsavailableatScienceDirect

Journal

of

Neuroscience

Methods

j ou rn a l h om epa g e : w w w . e l s e v i e r . c o m / l o c a t e / j n e u m e t h

Computational

Neuroscience

Cluster-based

computational

methods

for

mass

univariate

analyses

of

event-related

brain

potentials/ﬁelds:

A

simulation

study

C.R.

Pernet

a,∗

_,

_M.

_Latinus

b

_,

_T.E.

_Nichols

c

_,

_G.A.

_Rousselet

d a_Centre_for_Clinical_Brain_Sciences,_Neuroimaging_Sciences,_University_of_Edinburgh,_Edinburgh,_UK b_Institut_de_{Neurosciences}_de_la_Timone_UMR_7289,_Aix_Marseille_Université,_CNRS,₁₃₃₈₅_Marseille,_France c_Department_of_Statistics,_Warwick_University,_Coventry,_UK

d_Institute_of_Neuroscience_and_Psychology,_University_of_Glasgow,_Glasgow,_UK

a

r

t

i

c

l

e

i

n

f

o

Articlehistory:

Received26May2014

Receivedinrevisedform16July2014 Accepted5August2014

Availableonlinexxx

Keywords:

ERP

Family-wiseerrorrate Multiplecomparisoncorrection Cluster-basedstatistics

Thresholdfreeclusterenhancement Monte-Carlosimulations

a

b

s

t

r

a

c

t

Background:Inrecentyears,analysesofeventrelatedpotentials/ﬁeldshavemovedfromtheselectionof afewcomponentsandpeakstoamass-univariateapproachinwhichthewholedataspaceisanalyzed. Suchextensivetestingincreasesthenumberoffalsepositivesandcorrectionformultiplecomparisons isneeded.

Method:Herewereviewallcluster-basedcorrectionformultiplecomparisonmethods(cluster-height, cluster-size,cluster-mass,andthresholdfreeclusterenhancement–TFCE),inconjunctionwithtwo computationalapproaches(permutationandbootstrap).

Results:DatadrivenMonte-Carlosimulationscomparingtwoconditionswithinsubjects(twosample Student’st-test)showedthat,onaverage,allcluster-basedmethodsusingpermutationorbootstrap alikecontrolwellthefamily-wiseerrorrate(FWER),withafewcaveats.

Conclusions:(i)Aminimumof800iterationsarenecessarytoobtainstableresults;(ii)below50trials, bootstrapmethodsaretooconservative;(iii)forlowcriticalfamily-wiseerrorrates(e.g.p=1%), permuta-tionscanbetooliberal;(iv)TFCEcontrolsbestthetype1errorratewithanattenuatedextentparameter (i.e.power<1).

1. Introduction

Event-relatedpotentials (ERP) and magnetic fields(ERF) are measurablecorticalresponsestoeventsusedtotrack cognitive processes. In a given experiment, theyare observable at mul-tiple locations in space (electrodes or magnetic field sensors) andtime.ERPandERFarecharacterizedbyvariouscomponents whicharestereotypicfeatures suchasa peakortroughat par-ticularlatencies.1 _While_for_decades_researchers_have_focused_on analyzingsuchspecificcomponents,recenttoolshavebeen devel-oped to analyze simultaneously the whole data space using a mass-univariateapproach,wherebystatisticaltestsareperformed ateverylocation and timepoint (e.g.Kiebeland Friston,2004; Oostenveldet al.,2011; Pernet etal., 2011).Thisapproach has themeritof notchoosinglocationsorcomponentsa prioriand

∗_{Corresponding}_author_at:_Centre_for_Clinical_Brain_Sciences_(CCBS), Neuroimag-ingSciences,TheUniversityofEdinburgh,Chancellor’sBuilding,RoomGU426D,49 LittleFranceCrescent,EdinburghEH164SB,UK.Tel.:+441314659530.

E-mailaddress:[email protected](C.R.Pernet).

1_{http://www.sinauer.com/fmri2e/html/glossary.html}_.

therefore allows to potentially observing non-expected effects. Becausesomanystatisticaltestsareperformed,suchapproachcan dramaticallyincreasetheoddsofobtainingsigniﬁcanteffects,i.e. thereis ahighprobabilityoffalsepositive results(type1 error rate).Fortunately,differentmethodsexisttocontrolthe family-wiseerrorrate(FWER),i.e.thetype1errorrateoveranensemble, orfamily,oftests.Thetype1FWERisdeﬁnedastheprobabilityto makeatleastonetype1erroroverthefamilyoftests.Probablythe bestknownmethodtocontroltheFWERistheBonferroni correc-tion(Dunn,1961)forwhichthealphalevelissimplyadjustedfor thenumberoftests.Thismethodishoweveroverlyconservative inthecontextofERP/ERFanalysesbecauseitassumesstatistical independenceofthetests.ForERPandERF,therearealarge num-berofdependenciesinspaceandintime,suchthatstatisticaltests arenotindependent.Methodsusedtocontrolthetype1FWER insuchcontextmustthereforeaccountforthesespatiotemporal dependencies.

ERPandERFaredistributedsignals.Becausethereareapriori effectseverywhere,it iscommonpracticetodiscretizethedata spaceanddeﬁnetreatmenteffects.Suchdiscretizationleadstothe examinationoftreatmenteffectsintermsoftopologicalfeatures

http://dx.doi.org/10.1016/j.jneumeth.2014.08.003

(3)

2 C.R.Pernetetal./JournalofNeuroscienceMethodsxxx(2014)xxx–xxx

(4)

likethemaximum (e.g.+2␮V)ortheextent(e.g.from+120ms to+190ms poststimulusonset)of theeffect.In turn,thisdata reduction diminishes themultiple comparisons problem, while accountingforspatiotemporaldependences.Onepopularmethod thatdealswithmultiplecomparisonsbytakingintoaccountthe topologyoftheeffectsisrandomﬁeldtheory(WorsleyandFriston, 1995).AlthoughitwasdevelopedforPositronEmission Tomogra-phy(PET)andfunctionalMagneticResonanceImaging(fMRI),ithas alsobeensuccessfullyappliedtoElectroEncephaloGraphy(EEG) and MagnetoEncephaloGraphy(MEG) data(Kilneret al., 2005). Similarly,extensionsoftheBonferronicorrectionfor dependent datahavebeenproposed(Hochberg,1988).Therearecurrently nolargesimulationresultsontheapplicationofthesemethods toERP/ERF,but previousworkonrealand simulatedfMRI sta-tistical imagessuggeststhat theyare tooconservative(Nichols andHayasaka,2003).Inadditionthosemethodsalsorelyon var-iousassumptionslikepositivedependenceorsmoothness. Here wechoosetoreviewalternativemethodsthat combine cluster-basedinferencewithassumption-freetechniqueslikepermutation, whichhavebeenshowntooutperformanalytictechniquesto con-trolthetype1FWER(NicholsandHayasaka,2003).

Cluster-basedstatisticsconsistingroupingtogether neighbor-ingvariables(torFvaluesforinstance)intoclustersandderiving characteristicvaluesfortheclusters.Typically,aclusteris char-acterizedby its height(maximal value), its extent (number of elements)oracombinationofboth(PolineandMazoyer,1993). Inthelastcase,thisisoftenobtainedbysummingthestatistical valueswithinacluster,anapproachreferred toascluster-mass (Bullmoreetal.,1999;MarisandOostenveld,2007–seeFig.1for anillustration).Afirstissuewithtraditionalclusterinferenceisthat clustersneedtobedefinedbysettinga‘clusterformingthreshold’ (Gorgolewskietal.,2012).Inpractice,statisticalvaluesare consid-eredforinclusioninaclusteronlyiftheyarehigherthanthecluster formingthreshold,forinstanceaunivariatep<0.05.Itisthen pos-sibletocomputeclusters’attributesandtheirassociated probabili-ties.Asecondissuewithclusterstatisticsisthatinferencesare lim-itedtoclusters,i.e.onecannotbecertainofthesignificanceofsingle elementsinsideclusters.Morerecently,SmithandNichols(2009)

haveproposedto‘enhance’torFvalues,byintegratingattributes (heightandextent)computedforallpossibleaprioricluster form-ingthresholds(Eq.(1)),leadingtostatisticalmapswhereeachdata point,ratherthancluster,canbethresholded.Thismethod,referred toasThresholdFreeClusterEnhancement(TFCE),hasthe advan-tageofalleviatingissuesofsettingaclusterformingthresholdand ofclusterinferenceandhasbeenshowntocontrolthetype1FWER forERPusingpermutation(MensenandKhatami,2013).

TFCE (loc,time)=

h(loc,time)

h=h0

extent (h)Eheight (h)Hdh (1)

TheTFCEvalueatagivenlocation(loc)andtimepoint(time)is theintegralofallcluster-extents×cluster-heightsfromh0 (typi-callytheminimumvalueinthedata)toh(typicallythemaximum valueinthedata).ParametersEandHaresetto0.5and2 respec-tivelyinLIMOEEG(thesechoicesarediscussedbelow).Inpractice theintegralisestimatedasasum,usingﬁnitedh(heredh=0.1).As discussedbySmithandNichols(2009),TFCEisageneralizationof thecluster-massstatistic(E=1,H=0),andcanberelatedtoRandom FieldTheoryclusterp-values.

Inthepresentstudy,usingdatadrivensimulations,we evalu-atedtheabilityofcluster-basedcomputationalmethodstocontrol

thetype1FWER.Theﬁrstgoalofthestudywastoestablishthe equivalenceofpermutationandbootstrapprocedurestocontrol thetype1FWER,inthecontextofcluster-massforERP. Cluster-massisthemethodimplementedinbothLIMOEEG(Pernetetal., 2011;https://gforge.dcn.ed.ac.uk/gf/project/limoeeg/)and Field-Trip(Maris andOostenveld,2007;http://ﬁeldtrip.fcdonders.nl/). LIMOEEGusesabootstrap-ttechnique,and FieldTripuses per-mutation,butthetwotechniqueshavenotbeencomparedinthis context.Inaddition,cluster-masswasonlyvalidated,inFieldTrip, fortime–frequencydata.Thesecondgoalofthisstudywasto vali-date theTFCEmethodfor ERP,usingthebootstrap-t technique implementedinLIMOEEG,asopposedtopermutationasinMensen andKhatami(2013).

2. Methods

Codes used to generate and analyze the data are avail-able on FigShare at http://ﬁgshare.com/articles/Type1error rateusingclusteringforERP/1008311.Thedatageneratedto com-pareclusteringapproaches are toolargetobeshared(_∼32GB) but the code could be used on any data and similar results are expected. Intermediate results (i.e. FWER data per sub-ject)areneverthelessavailable.For theTFCEsimulations,codes and intermediate dataare alsoavailable at http://ﬁgshare.com/ articles/Type1errorrateusingclusteringforERP/1008325.

2.1. Cluster-attributesusingbootstrapandpermutation

Whilstsimulationsaimedatcomparingbootstrapand permu-tationtechniquesinthecontextofcluster-mass,wealsocomputed cluster-extentandcluster-heighttoenquirepotentialdifferences. Simulationswereperformedinthecontextofawithinsubjecttwo sampleStudent’st-test,comparingtwohypotheticalconditions, butresultsapplyinprincipletootherwithinsubjectcases.

TensubjectsoftheLIMOEEGdataset(Rousseletetal.,2009; Rousseletet al.,2010)wererandomlychosenasrepresentative ERPdata.Datawerefroma128electrodesBiosemisystem,250Hz samplingrate,with201timepointepochsrangingfrom−300ms to500ms.Duringrecording,subjectsdiscriminatedbetweentwo faceswithvariouslevelsofnoise(seereferencesfordetails)and performedover1000trials.OnethousandMonteCarlo(MC) sim-ulationswereperformedpersubject(10,000MCintotal).Foreach MonteCarloandeachsubject,samplesof10,25,50,100,300,500, 900trialspergroupswereobtainedbyincrement,e.g.whenN=25, thesame10trialsaswithN=10werepresent.ForeachMC,data wererandomlyassignedtoacondition(sayfaceAversusfaceB) andatwosampleStudent’st-testcomputedforeveryelectrode andtimepoint.Threetechniqueswereusedtoestimatethenull hypothesis (H0)for thesampleconsidered: (i)a permutation t-testinwhichallthetrialsfromthetwoconditionsarepermuted randomlybetweenconditionsandat-testiscomputedforeach permutation,(ii)amodifiedpercentilebootstrapinwhichallthe tri-alsfromthetwoconditionsarefirstpooledtogether,thensampled withreplacementand randomlyassigned tothetwoconditions and a t-test computed for each bootstrap, (iii) a bootstrap-t in whicheachconditionisfirstmeancentered,thensampledwith replacementandat-testiscomputedforeachbootstrap.For all three techniques,1000 iterationswere computedand maximal valuesrecordedusingcluster-formingthresholdsofp=0.05and p=0.01.Intotal7000drawsofdatawerecomputedpersubject

(5)

(1000MC×7samplesizes)and42,000statisticalmapsobtained (7000draws×3techniques×2cluster-formingthresholds).

2.1.1. Exploratorydataanalysis

For each subject, technique, cluster-forming threshold, and samplesize,wecomputedthenulldistributionsofmaximaforthe3 clusterstatistics:cluster-height,clusterextent,andt2_cluster-mass

andthresholdedthesampleddataaccordingtothesedistributions. Thetype1FWERforthecriticalFWEthresholdsof p=0.05 and p=0.01wascomputedastheprobabilitytoobtainatleastone sig-niﬁcanteffectacrossallelectrodesandtimepointsoverthe1000 MCandthenaveragedoversubjects.Deviationfromthesettype 1errorratewastestedforeachcombinationofclusterstatistics andtechnique(e.g.cluster-mass/permutations)bycomputing per-centilebootstrap95%conﬁdenceintervals(CI)withaBonferroni adjustmentforsimultaneousprobabilitycoverageoverthe7 sam-plesizes(i.e.alpha=0.9929%–Wilcox,2012).Wealsocompared thepercentage of agreementbetweenthedifferenttechniques, whichistheproportionoftimesthesameresultswereobserved outof10×1000MC.Finally,welookedathowmanypermutations orbootstrapswerenecessary(from200to1000bystepsof200)to achievethenominalFWER.

2.1.2. Conﬁrmatorydataanalysis

Totesttheequivalenceofpermutationandthetwobootstrap proceduresforcluster-massinference,simulationresultswere col-lapsedover allsample sizesand apercentilebootstrap (with a Bonferroniadjustmentformultiplecomparisons)wascomputed onthemean,20%trimmedmean,andmediantype1FWER, test-ingifthenominallevelwasobtained.Pair-wisecomparisonsofthe differenttechniques(permutationvs.percentilebootstrap, permu-tationvs.bootstraptandpercentilebootstrapvs.bootstrapt)were alsoperformedusingapercentilebootstraponthemean differ-ences.

2.2. TFCEvalidation

Adifferentsetof datadriven Monte Carlosimulationswere performedonthesame10 subjectsas above.For each subject, 1000 MC samples of 200 trials, 100 trials per condition, were drawnrandomlyandwithreplacementfromapoolofover1000 trials,thusmimickingaseriesofdrawsfromthesamepopulation. ForeachMCsample,atwo-sampleStudent’st-testwascomputed. Foreachoftheset-tests,thenullhypothesiswasevaluatedusing a bootstrap-t technique with 1000 iterations. The observed t values were thresholded using cluster-mass (cluster-forming threshold p=0.05) and using TFCE. Since TFCE integrates mul-tiplethresholded maps,theclusterextentand heightcan have differentpowers leadingto differentenhanced values (Eq.(1)). Here we tested 4 possible combinations of TFCE parameters:

extent ˆ0.5*height ˆ1, extent ˆ0.5*height ˆ2, extent ˆ1*height ˆ1 and extent ˆ1*height ˆ2.Foreachsubject,the5000maps(1000MCfor cluster-mass+1000MC×4 TFCE) werethresholded ata critical 5% FWE thresholdand thetype 1 FWER computed.The mean, 20% trimmed mean, and median FWER across subjectsfor the 4 combinationsof TFCEparameters werecomputed and tested againstthenominallevel(percentilebootstrapwithalphaadjusted forsimultaneousprobabilitycoverageoverthe4possible combi-nations)andcomparedwithcluster-mass(bootstrap-twithalpha adjustedformultiplecomparisons).

3. Results

3.1. Exploratorydataanalysis

Resultsshowthatthesixcombinationsoftechniques (permuta-tion,bootstrap)andclusterstatistics(cluster-mass,cluster-extent, cluster-height)controlledwellthetype1FWER(Figs.2and3).In addition,weobservedahighpercentageofagreementbetween sta-tisticalmasks(always>99%)demonstratingtheequivalenceofthe techniques.Theonlycaseswherethetype1FWERdeviatedfrom thenominallevelwaswiththesmallestsamplesizes(N=10or 25trialspercondition),wherebootstrapsweretooconservative. Oneexceptionwasobservedforcluster-heightunderpercentile bootstrapwithacritical5%FWER,whichgave atoohighvalue. ToachievethenominalFWER(Figs.4and5), between600and 800iterationswerenecessary,irrespectiveofthetechnique con-sidered.Iffewerthan600iterationsweredrawn,resultsweretoo liberal(exceptagainforbootstraptechniqueswithsmallsample sizeswhichwerealwaystooconservative).

3.2. Conﬁrmatorydataanalysis

Foracritical5%FWER,onaverageacrosssamplesizes, cluster-massinferenceforthethreetechniquesgivesaFWERthatdidnot differsignificantlyfrom5%(Table1).Significantmeandifferences wereneverthelessobservedamongthethreetechniques,with per-mutation showing systematically higher FWER than bootstrap: permutationvs.percentilebootstrap0.0016[0.0004,0.0027]p=0; permutation vs. bootstrap-t 0.0022 [0.0011, 0.0035] p=0; per-centilebootstrapvs.bootstrap-t0.0006[−0.0003,0.0015]p=0.09. Foracritical1%FWER,onaverageacrosssamplesizes, cluster-massinferenceforthethreetechniquesgivesaFWERclosetothe nominallevel,withsignificantdeviationinthecaseof permuta-tions,beingtooliberal(Table2).ThemeanFWERwassignificantly higher for permutations compared to the percentile bootstrap (0.007[0.0001,0.0012]p=0.006),butnottothebootstrap-t(0.0042 [−0.0002,0.001]p=0.09).Thepercentilebootstrapdidnotdiffer significantlyfrombootstrap-t(−0.0028[−0.0009,0.0003]p=0.34).

Table1

Mean,20%trimmedmeanandmedianofthecluster-massFWERfora5%criticalthreshold.Inbracketsaretheadjusted95%CI.

Permutation Percentilebootstrap Bootstrap-t

Mean 0.0517[0.0498,0.0536] 0.0501[0.0479,0.0523] 0.0495[0.0471,0.0519] 20%Trimmedmean 0.0517[0.0497,0.0540] 0.0497[0.0476,0.0524] 0.0493[0.0469,0.0519] Median 0.0515[0.049,0.054] 0.0495[0.048,0.0515] 0.049[0.047,0.0515]

Table2

Mean,20%trimmedmeanandmedianofthecluster-massFWERfora1%criticalthreshold.Inbracketsaretheadjusted95%CI.Signiﬁcantdeviationsfromthenominallevel areinbold.

Permutation Percentilebootstrap Bootstrapt

(6)

(7)

(8)

Fig.4. Type1FWERforacritical5%FWE(clusterformingthresholdp=0.05)foreachclusterstatisticandtechniqueasafunctionofthenumberofsamplingiterationsfor the7samplesizestested(n=[102550100300500900]pergroup).

(9)

Fig.6.Type1FWERforcluster-mass(CM)andThresholdFreeClusterEnhancement(TFCE)using4combinationsofextentandheight.Boxesshowforeachsubjectthemean type1FWERandassociatedbinomial95%CI.Thebottomrightplotsshowbargraphsofthemeantype1FWERacrosssubjectsand95%CI,andthemeandifferencesbetween eachTFCEparametersetandcluster-masstype1FWER.

Table3

TFCEmeantype1FWER(critical5%FWER)andmeandifferencesbetweenTFCEand cluster-mass.Theadjusted95%CIareindicatedinsquarebrackets,andsigniﬁcant deviationsareinbold.

MeanFWER Differencetocluster-mass

Cluster-mass 0.0457[0.0392,0.0522]

TFCEEˆ0.5*Hˆ1 0.0487[0.0435,0.0539] −0.0030[−0.0053,−0.0008]

TFCEEˆ0.5*Hˆ2 0.0478[0.0423,0.0533] −0.0021[−0.0057,0.0017] TFCEEˆ1*Hˆ1 0.0459[0.0423,0.0495] −0.0002[−0.0058,0.0044] TFCEEˆ1*Hˆ2 0.0444[0.0403,0.0485] 0.0013[−0.0020,0.0045]

3.3. TFCEvalidation

Thepercentilebootstrapt-testwithsimultaneous95% probabil-itycoverageshowedthatTCFEwiththeextentparameterofpower 1wastooconservative(Table3,Fig.6).Comparisonswith cluster-massshowthattheparametercombinationextent ˆ0.5*height ˆ1was signiﬁcantlylessconservativethancluster-mass,whileother com-binationsshowresultssimilartocluster-mass.

4. Discussion

Overall,oursimulationsshowthat cluster-basedapproaches providea type 1 FWER closeto orat thenominal level.Three exceptionswereneverthelessobserved:(i)cluster-statisticswith permutationcantobetooliberal;(ii)cluster-statisticswith boot-straptechniquesandsamplesizesof10and25pergrouparetoo conservative;(iii)TFCEwithextentparametersatpower1aretoo conservative.Withregardstothestudymaingoalsweshowedthat (i)permutationand bootstraptechniquesgive,onaverage, very similarresultsandthat(ii)TFCE,inconjunctionwithbootstrap, canbeavalidmethodforERPinference.

Whilesomedeviationsareexpectedbetweentechniques, boot-strapshowedstrongdeviationsforsmallsamplesizes,whichcan

beexplainedbythehighclusterstatisticvaluesobtainedunderH0. Toillustrate,letusconsidercluster-masswithacritical5%FWER thresholdandN=10:overthe10×1000MonteCarlosimulations, permutationgavearangeofcluster-massthresholdsfrom1390to 24,300(median18,321),whereasthepercentilebootstrapgavea rangeofcluster-massthresholdsfrom17,843to31,594(median 22,954), andthebootstrap-t a range ofcluster-mass thresholds from21,096to44,036(median29,190).Onereasonforthe ele-vatedbootstrapthresholdscouldrelatetothesamplingscheme.A t-testisdeﬁnedbytheratiobetweenthemeandifferenceandthe squarerootofthesumofstandardizedvariances.Duringbootstrap resampling,ifonlyfewuniquetrialsaredrawn,thiscanreducethe variancetoapointwherethedenominatorisinferiorto1,whichin turncanleadtolargetvalues.Inthesimulations,wedidnot con-straintthenumberofuniquetrials,andre-runningthesampling methodtogenerateindicesindicatesthat,foragivensubject,no uniquevaluesweredrawn,butaslowas2uniquetrialswereused whichwouldhaveledtohightvalues.Permutationisnotaffected bythisissuebecausethesamenumberofdifferenttrialsispresent ateachiteration,maintainingvarianceatareasonablelevel.For smallsamplesizes,itisthusrecommendedtouseapermutation test,ortoconstrainttheminimumnumberofuniqueobservations inthebootstrapsamples.Ofcourse,whatevertechniqueisused, statisticalinferencesarefundamentallylimitedwhenonly10trials areused.

(10)

800iterations:in onesubject,1000MCusingabootstrap-ttest withN=10showednochangesintype1errorratebetween800 iterationsandupto3000iterations(type1FWER=0.019).

WevalidatedtheuseofTFCEinconjunctionwithabootstrap-t techniqueinthecontextofERPanalyses,andshowedthatusing E=0.5(i.e.extent ˆ0.5inEq.(1)),thetype1errorrateisatthe nom-inallevel.UsingE=1gavetooconservativeresults.Underthenull hypothesis,thisislikelyduetolargeclusterswhenhissmall,which, onceintegratedleadtohighTFCEthresholds,whichinturnleadto aconservativetype1FWER.AspointedoutbySmithandNichols (2009),atthelowestvaluesofh,thesigniﬁcantclustersaretoolarge anddonotprovideveryusefulspatiotemporalspeciﬁcity,anditis thereforepreferabletoscaledowntheireffect.Conversely,forthe parameterH,whenH>1,theTFCEscoresscalesupra-linearlywith increasingstatisticimageintensity,whichcanbeconsidertobe desirable.WethusfollowSmithandNicholsandalsosuggestto useH=2(i.e.height ˆ2inEq.(1)),sincesquaringfollowsthelogof p-values(seeSmithandNichols,2009)andgivesresultssimilarto cluster-massinoursimulations.MensenandKhatami(2013) val-idatedTFCEforEEGusingpermutation,withparametersE=2/3, E=1andH=2.Intheirsimulationsthedatahadbothnoeffect(H0) andsomeeffects(H1),andtheycomputedthebalancebetween type1andtype2errorrates.ThebestresultwasobtainwithE=1. GiventhattheydidnottesttheFWERoverthewholespace,we believethatE<1remainsthebestoptiontoachievethenominal FWER.

Thetype1FWERwasestimatedwithrealnulldataand there-foretherearenoissuesofsignal-to-noiseratio(SNR),ornumber ofsources.Inaddition,becausethemethodsdescribedhereadapt totheacquisitionparametersbyestimatingthenullhypotheses fromthedatathemselves,aspectsofdataacquisitionrelatedto sampling(samplingrateornumberofelectrodes)shouldnotaffect theinterpretationofourresults.Inthistypeofsimulation,only thevariance–covariancestructurehasaninﬂuenceontheresults. Althoughthelargescale structureof thecovariance matrix(i.e. which electrodes correlate with which, when) depends onthe experimentalset-up,thesmallerscalestructureofthecovariance matrix(i.e.howtrialscorrelate)isexpectedtobesimilaracross datasets.Wecanthereforeexpecttoobtainsimilarresultsusing otherdatasets,andwedoremindinterestedreadersthatthecode is availabletotry ontheirown data.In contrastto thetype 1 FWERtestedhere,manyparameterswillaffectpower.The abil-itytodetectaneffectanddeclarethiseffectassigniﬁcantislikely todependontheSNR,thespatiotemporalsampling,andthe statis-ticalmethodused.Inparticular,clusterextentandclusterheight willbeaffectedbydifferencesinsourcedepthandSNRanditis recommendedtousecluster-massorTFCEinmostsituations.

Inconclusionweshowedthatpermutationandbootstrap tech-niquescan beusedalike in combinationwithcluster statistics, includingTFCE,providingaccuratetype1FWER.Inthecaseofsmall samplesizes(N=10,25trials),itisadvisabletousepermutationas itoffersabettercontroloverthetype1FWER,whereasbootstrap techniquesaretooconservative.Conversely,forlargersamplesizes

(N≥50),itisadvisabletousebootstraptechniquesbecause permu-tationcanbetooliberal.Finally,althoughoursimulationsaimedto showgenerallyapplicableresults,itremainstobetestedhowthese techniquesbehavewithdatahavingdifferentvariancestructures. Forinstance,betweensubjectanalysestendtohavesmallsamples withlargevariances,whichcanbeaproblemforpermutationifthe dataareheteroscedastic(EfronandTibshirani,1993).

Acknowledgment

ThisworkissupportedbytheBBSRCgrantsBB/K014218/1(GAR) andBB/K01420X/1(CRP).

References

BullmoreET,SucklingJ,OvermeyerS,Rabe-HeskethS,TaylorE,BrammerMJ.Global, voxel,andclustertests,bytheoryandpermutation,foradifferencebetween twogroupsofstructuralMRimagesofthebrain.IEEETransMedImaging 1999;18(1):32–42.

DunnOJ.Multiplecomparisonsamongmeans.JAmStatAssoc1961;56:52–64.

EfronB,TibshiraniRJ.Anintroductiontothebootstrap.NewYork:Chapmanand Hall;1993.

GorgolewskiKJ,StorkeyAJ,BastinME,PernetCR.Adaptivethresholdingfor reli-abletopologicalinferenceinsinglesubjectfMRIanalysis.FrontHumNeurosci 2012;6.,http://dx.doi.org/10.3389/fnhum.2012.00245.

HochbergY. AsharperBonferroniprocedureformultipletestsofsigniﬁcance. Biometrika1988;75(4):800–2.

KiebelSJ,FristonKJ.Statisticalparametricmappingforevent-relatedpotentials:I. Genericconsiderations.Neuroimage2004;22(2):492–502.

KilnerJM,KiebelSJ,FristonKJ.Applicationsofrandomﬁeldtheoryto electrophysi-ology.NeurosciLett2005;374(3):174–8.

MarisE,OostenveldR.NonparametricstatisticaltestingofEEG-andMEG-data.J NeurosciMethods2007;164(1):177–90.

Mensen A, Khatami R. Advanced EEG analysis using threshold-free cluster-enhancement and non-parametric statistics. Neuroimage 2013;67: 111–8.

NicholsT,HayasakaS.Controllingthefamilywiseerrorrateinfunctional neuroimag-ing:acomparativereview.StatMethodsMedRes2003;12(5):419–46.

OostenveldR,FriesP,MarisE,SchoffelenJM.FieldTrip:opensourcesoftwarefor advancedanalysisofMEG,EEG,andinvasiveelectrophysiologicaldata.Comput IntellNeurosci2011;201:1,http://dx.doi.org/10.1155/2011/156869. PernetCR,ChauveauN,GasparC,RousseletGA.LIMOEEG:atoolboxfor

hierar-chicallinearmodelingofelectroencephalographicdata.ComputIntellNeurosci 2011;2011:1–11,http://dx.doi.org/10.1155/2011/831409.

PolineJ-B,MazoyerBM.Analysisofindividualpositronemissiontomography acti-vationmapsbydetectionofhighsignal-to-noise-ratiopixelclusters.JCereb BloodFlowMetab1993;13(3):425–37.

RousseletGA,GasparCM,PernetCR,HuskJS,BennettPJ,SekulerAB.Healthyaging delaysscalpEEGsensitivitytonoiseinafacediscriminationtask.FrontPsychol 2010.,http://dx.doi.org/10.3389/fpsyg.2010.00019.

Rousselet GA, Husk JS, Pernet CR, Gaspar CM, Bennett PJ, Sekuler AB. Age-related delay in information accrual for faces: evidence from a parametric, single-trial EEG approach. BMC Neurosci 2009;10(1):114,

http://dx.doi.org/10.1186/1471-2202-10-114.

Smith S, Nichols T. Threshold-free cluster enhancement: addressing problems of smoothing, threshold dependence and local-isation in cluster inference. Neuroimage 2009;44(1):83–98,

http://dx.doi.org/10.1016/j.neuroimage.2008.03.061.

TroendleJ,KornE,McshaneLM.Anexampleofslowconvergenceofthebootstrap inhighdimensions.AmStat2004;58(1):25–9.

WilcoxR.Introductiontorobustestimationandhypothesistesting.3rded.Oxford, UK:AcademicPress,Elsevier;2012.