Validating maps from single particle electron cryomicroscopy

Validating

maps

from

single

particle

electron

cryomicroscopy

Peter

B

Rosenthal

1

and

John

L

Rubinstein

2,3,4

Progressinsingleparticlecryo-EM,mostrecentlyduetothe introductionofdirectdetectordevices,hasmadethe high-resolutionstructuredeterminationofbiologicalassemblies smallerthan500kDamoreroutine,buthasalsoincreased attentionontheneedfortoolstodemonstratethevalidityofsingle particlemaps.Althoughmapvalidationisacontinuingsubjectof research,someconsensushasbeenreachedonproceduresthat reducemodelbiasandover-fittingduringmaprefinementaswell asspecificteststhatdemonstratemapvalidity.Tilt-pairanalysis maybeusedasamethodfordemonstratingtheconsistencyat lowresolutionofamapwithimagedata.Forhigher-resolution maps,newproceduresformorerobustresolutionassessment andforvalidatingtherefinementofatomiccoordinatemodels intosingleparticlemapshavebeendeveloped.

Addresses

1_{FrancisCrickInstitute,MillHillLaboratory,LondonNW71AA,United}

Kingdom

2

MolecularStructureandFunctionProgram,TheHospitalforSick

ChildrenResearchInstitute,Toronto,OntarioM5G1X8,Canada

3_{DepartmentofBiochemistry,TheUniversityofToronto,Toronto,}

OntarioM5S1A8,Canada

4

DepartmentofMedicalBiophysics,TheUniversityofToronto,Toronto,

OntarioM5G1L7,Canada

Correspondingauthors:Rosenthal,PeterB

([email protected])andRubinstein,JohnL ([email protected])

CurrentOpinioninStructuralBiology2015,34:135–144

ThisreviewcomesfromathemedissueonBiophysicaland

molecularbiologicalmethods

EditedbyBenSchulerandJanetLSmith

ForacompleteoverviewseetheIssueandtheEditorial

http://dx.doi.org/10.1016/j.sbi.2015.07.002

0959-440/#2015TheAuthors.PublishedbyElsevierLtd.Thisisan

openaccessarticleundertheCCBY-NC-NDlicense(

http://creative-commons.org/licenses/by-nc-nd/4.0/).

Introduction

Recentdevelopmentsinbothexperimentaland

compu-tationalaspectsofsingleparticleelectroncryomicroscopy

(cryo-EM)havenowfulfilledpredictionsthatthemethod

canbeusedforatomicresolutionstructuredetermination

of protein and nucleic acid assemblies [1] by aligning

and averagingimagesofnon-crystallinesingle particles.

Iterative refinement is typically used to bring a

low-resolutionstartingmapintohigherresolutionagreement

withstructuralsignalintheimages(seeBox1).However,

model-based assignment of particle orientations in low

signal-to-noise ratio, low-dose images can be incorrect

when map projections match the wrong features in an

image orwhentheymatchnoise.Thepersistenceof an

incorrectmapormapfeaturesduringrefinement,referred

to asmodelbias,isarealdanger.Forexample,acrystal

structureorhomologymodelusedasaninitialreference

during iterative refinement may yield a final map that

shows high-resolution features such as a-helices and

aminoacidside-chains,butinfactonlycontains

informa-tionpresent intheinitialreferencestructureandnotin

the images [2,3–5]. A related problem, termed

over-refinementorover-fitting,occurswhenthemapmatches

signalintheimagesatlowresolution,butathigh

resolu-tionpredominantlymatchesnoise[6,7,8].Theresultof

thisover-fittingisthebuild-upofspuriousnoisefeatures

inthemapthatarereinforcedduringiterativerefinement.

Over-refined maps can possess features that resemble

side-chaindensities butin incorrectlocations[9].

AtasimilarstageinthedevelopmentofX-ray

crystallog-raphy,theproblemofover-fittingofmodelsduring

refine-ment was addressed by excluding a small fraction of

reflectiondata(‘freeset’)fromthedatausedinrefinement

(‘workingset’) forcross-validation.The ‘Rwork’,the

re-sidualerrorbetweentheworkingsetandreflections

calcu-latedfromthemodel,canbedriventoarbitrarilylowvalues

byincludingmorerefinementparameters,evenwhenthe

modelpoorlymatchesthedata,butdecreaseofthe‘Rfree’,

the residual error between the free set and reflections

calculated from the model, indicates true improvement

inthemodel[10].Inrecentyears,progresshasbeenmade

insingleparticlerefinementschemes,includingtheuseof

maximum likelihood strategies thatare less sensitiveto

modelbiasandover-fittingthanearlierapproaches[4,11–

13].Inaddition,specificvalidationtoolshavebeen

devel-opedandtheimportanceofapplyingandreportingthem

during publication and database deposition has been

agreedupon[14].Below we describeessential

princi-plestoensurethatsingleparticlemapsagreewithdataat

low resolution and that the resolution of the structure

determinationiscorrectly assessed,particularlyathigh

resolution.Weemphasizespecificteststhatdemonstrate

thevalidityofamapwithouttherequirementfor

inde-pendentre-determinationofthemapfromrawdata.We

also describe similar principles and tests that may

beappliedtovalidatingatomicmodelsbuiltintosingle

particlemaps.

Is

a

map

correct

at

low

resolution?

Anincorrectstarting mapmaypersistduringrefinement

be applied from the very beginning of any structure

determination.Becauseimagescontainingonlynoisewill

align to the map during refinement, it is essential to

demonstratethatindividualparticlesofthereportedsize

and shape are actually present and visible in images.

Representative high contrast images may be obtained

byemployingalargeobjectivelensdefocusandsufficient

electron exposure [2]. Identification and selection of

particleimagesinmicrographsbycross-correlationwitha

templatemayselect pure noiseimages [4,5,15].In this

case,subsequentimageprocessingmaybebiasedbythe

template. Consequently, thesafeguardof visual

exami-nationofthemicrographsandaninitialroundofmanual

particle picking by the investigator should always be

employed.

Followingparticleselection,thenextimageprocessing

stepsare thealignment, classificationandaveraging of

images of particles in similar orientations to produce

highsignal-to-noiseviewsoftheparticle(classaverages).

Theindividual particleimages that comprise theclass

shouldresembletheclassaverage;thismaybeusedasa

criteriontoselectgoodparticleimages[16].

Subsequent-ly,theseclassaveragesmaybeassignedorientationsto

calculateaninitialmaportheymayserveasan

interme-diatestageiniterativemaprefinement.Reference-free

procedures[3,17]thatarelesssusceptibletonoise

match-ingatemplate,suchasalignmentofparticlestoa

rota-tionallyaveraged sum of particle images or maximum

likelihoodstrategies,shouldbeusedtocalculatethefirst

classaverages.Similarly,projectionsofthe3Dstructure

shouldagree withboth classaverages andrawimages.

Classaveragesthat donot representprojections ofthe

structurecanarisewhendifferentviewsofaparticleare

inappropriately averaged or when noise in an image

alignsto a2Dtemplate.Whenparticlesadopt random

orientations in an ice film, pairs of classaverages that

are mirror images of each other give support to those

class averages being true ‘views’ of a macromolecular

assembly[18]. Box1 PARTICLE STACKS MATCHING PROJECTIONS 3D MODEL ORIENTATION PARTICLE STACKS (−α) MATCHING PROJECTIONS MODEL-BASED DETERMINATION

OF ORIENTATION PARAMETERS NEW MODEL

STARTING MODEL PARTICLE STACKS (+α) MATCHING PROJECTIONS (b) (a) +α −α

Current Opinion in Structural Biology

Model-baseddeterminationoforientationparametersfromsingleparticleimagesandtilt-pairs.(a)Projectionsofalow-resolutionstartingmodel

arematchedforagreementwithastackofparticleimages.Eachimageisthusassignedorientationparameterscorrespondingtothe3Dmodel

orientationthatmatchestheimage.Inaniterativerefinementscheme,anewmodeliscomputedfromtheimagesandtheimprovedmodelisused

todeterminemoreaccurateorientationparameters.(b)Imagesofasinglefieldofparticlesarerecordedattwoangles( aand+a)ofthe

microscopegoniometerandarecalledatilt-pair.A3Dmapisusedtodetermineorientationparametersforbothparticlestacksbyprojection

Furthermore, 3Dstructure determinationwith isotropic

resolution requiresawidedistributionof particleviews,

althoughthereareanumberofwaysinwhichanadequate

distribution may be obtained [19]. The distribution of

Euler angles assigned to particle images should be

reported.Somemap calculation algorithmscanproduce

artefacts if specific particle orientations are

over-repre-sented [20]. Point symmetry determines which views

should beaveraged,thesearchrangeusedtodetermine

particle orientations,andthenumberof particleimages

needed to achieve a given resolution. Compelling

evi-dence for viewsshowing symmetry consistent with the

point symmetryofaparticle[21,22]shouldbeobtained

beforethatpointsymmetryisappliedtothemap.Proofof

thesymmetry ultimatelyrequires validation ofthemap

byanumberofothertests,suchasthosedescribedbelow.

Alow-resolutionmapmayhaveapproximateagreement

withimagesorclassaveragesandstillbeincorrect.Amore

stringent criterionfor acorrectmap isthesimultaneous

agreement of a map with image data recorded at two

angles of the microscope goniometer. This agreement

demonstratestheconsistencyofthemapwithstructural

featuresoftheparticle,whichchangepredictablywithtilt

angle, showing that orientations are correctly assigned

andthatthemapisnotmerelymatchingnoise.The

tilt-pair test is a simple procedure for cross-validation of

orientationdeterminationrequiringa3Dmapandasmall

representativesampleofthedataintheformoftwostacks

ofparticleimages(e.g.100particles)fromthesamefield

ofparticlesrecordedattwodifferentrotationanglesofthe

microscope goniometer [23]. Tilt-pair analysis is now

implementedin several image processing packages and

asawebservice([24],http://www.ebi.ac.uk/pdbe/emdb/

validation/tiltpair/). This test can help detectthe worst

typeoferrorinwhichamapiscompletelyincorrecteven

atlow resolution.

Thetilt-pairtestshowsthattheorientationsassignedto

theparticlesintheimagesagreewiththeknowntiltangle

andaxisoftheexperiment.Plottingthetilt axisandtilt

angle foreachparticlepairin adataset,calculated from

assigned orientations, and with a specification of the

particle symmetry, yields a tilt-pair parameter plot

(TPPP) [23,25] (Figure 1a). A slightly different form

of analysis, thetilt-pairphase residualplot,isshownin

Figure1b.Thedifferencebetweenthescoreforthemap

of correcthandcomparedtooneofoppositehand

(free-hand test) isameasure of thesuccess in absolutehand

determination, animportantmeasureoftheinformation

contentofamap[23,24].Bothtypesoftilt-pairplotsmay

be applied to the optimization of particle orientation

determinationsuchasthechoiceofdataresolutionrange

used for orientation determination. The width of the

clustersontheTPPPisameasureofaccuracyin

orienta-tion determination, which is greater for molecules of

largermolecularweight.Thetiltangleoftheexperiment

should be chosen to be sufficientlylarge to distinguish

clusters of points around the applied tilt angle from

clusters around the origin, which may occur when the

map does not match the image data. The question of

whenaTPPPvalidatesamaporwhenitshowsthemapto

beincorrecthasbeendiscussedbyseveralinvestigators.A

simplemajoritycriterionhasbeenproposed[25]where

60% of particle tilt-pairparametersshould beclustered

aroundtheknowntiltangleandtiltaxis.Otherwise,the

reasons forthelackofagreementshouldbeassessedby

inspection of the tilt-pair data for processing errors or

specimenheterogeneity.Thesignificanceofthetiltpair

experiment may also be assessed against the expected

resultfor randomorientationsbythetilt-pairalignment

test[26](Figure1c,d).Morerecently theclusteringof

the TPPP has been described by the concentration

parameter (k) of the Fisher distribution [27]. A wide

distributionofviewsshouldbeusedinthetilt-pair test.

Selection of asubset of tilt-pair datathat validates the

map is an unfair application of the test and one that

distorts its measure of orientational accuracy. Tilt-pair

analysishasbeenusedtoassessparticlescoveringawide

rangeof shapes andsizesembeddedin ice.In negative

stain, particle flattening may cause the tilt-pair test to

haveaminimumatatiltanglelessthantheexperimental

angle.

A number of studieshave argued for thevalidation of

map refinementby demonstrating that multiple,

inde-pendentrefinementsofdifferentlow-resolutionstarting

maps all converge to the same structural features.

Although this observation is circumstantial evidence

thatamapiscorrect,itdoesnotexcludethepossibility

that thereisasystematicbias towardan incorrectmap

(e.g. when refinement cannot distinguish orientations

related byan approximatesymmetryofaparticleor is

performedusingareferencemapofill-definedhand,or

when image datasets lack essential views), and

inde-pendent validation criteria are still required to assess

theseresults.

Cryo-EMmapsandtheirinterpretationcanalsobe

vali-dated by biochemical experiments that localize a map

feature by adding or removing a subunit, addition of

covalently-linked polypeptide such as GFP [28] or

MBP [29], antibodies [30–32], or protein (e.g. avidin

[33])or electrondense labels[34].

Resolution

assessment

A valid assessment of resolution indicates theaccuracy

with which a structural model may be built and adds

confidence to functional interpretation. Validation of

resolution assessment is essential to detect over-fitting

and design refinement strategies that reduce spurious

features at high resolution. The observed featuresof a

mapshouldbeconsistentwiththeresolutionassessment

corrected for contrast loss at high resolution due to

imperfections in images and computational errors in

reconstructiondescribed byanoverallmaptemperature

factor. Contrastmay berestored by‘sharpening’ witha

negativetemperature factorand figure-of-merit

weight-ing[23,35,36].Thenumberofparticleimagesrequiredto

reach a given resolution should sensibly reflect image

qualityandtheknowncomputationalerrorinorientation

Figure1 Pairs of images (%) 0-5° 5-10° 15-20° 1 0 -1 5 ° 20-25° 25-30° 30 35-40° -3 5 ° 40-45° 45-50° 50 55-60° -5 5 ° 60-65° 65-70° 70 75-80° -7 5 ° 80-85° 85-90° 90 _95-100° -9 5 ° 100-105° 105-1 10° 1 15-120° 1 1 0 -1 1 5 ° 120-125° 12 130-135° 5 -1 3 0 ° 135-140° 140-145° 14 150-155° 5 -1 5 0 ° 155-160° 16 165-170° 0 -1 6 5 ° 170-175° 175-180° 0 5 10 15 20 25 30 Angular error

bovine ATP synthase

0-5° 5-10° 15-20° 10-15° 20-25° 25-30° 30-35° 35-40° 40-45° 45-50° 50-55° 55-60° 60-65° 65-70° 70-75° 75-80° 80-85° 85-90° 90-95° _95-100° 100-105° 105-110° 110-115° 115-120° 120-125° 125-130 130-135° ° 135-140° 140-145° 145-150° 150-155° 155-160° 160-165° 165-170° 170-175° 175-180° 0 5 10 15 20 25 30 Angular error T. thermophilus V/A-ATPase (a) (b) (c) (d) 0° 45° – 15° – 30° – 45° 15° 0° – 30° – 45° 15° 30° – 15° 45° 30° X Y

*

*

*

*

*

*

**

* *

*

*

TILTDIRECTION TILTDIRECTION TILTANGLE=30. TILTANGLE=40. 270 degrees 180 degrees 90 degrees 0 degrees

*

*

*

*

* *

_*

**

**

*** **

****

**

* **

*

Current Opinion in Structural Biology

Mapandorientationvalidationbytilt-pairanalysis.(a)Thetilt-pairparameterplot(TPPP)showsthein-planetiltaxisandangleforindividualimage

pairsofT.thermophilusV/A-typeATPaserecordedat 158and+158rotationsofthegoniometer.Particlesforwhichorientationsarecorrectly

determinedinbothimagesclusterattheknowntiltaxisandangleoftheexperiment.FigurefromRef.[25_{].(b)Thetilt-pairphaseresidualplotfor}

theV/A-typeATPaseshowstheaveragephaseresidualscoreonthesecondimageforeachpossiblein-planetiltaxisandtiltangleappliedtothe

orientationassignedtothefirstimage.Aminimum,markedwithastar,occursat(0,30),theknowntilttransformationusedintheexperiment.The

phaseresidualdifference(freehanddifference)between(0,30)and(0, 30)is14.98.FigurefromRef.[69].(c)Tilt-pairalignmenttestshowsthatthe

angularerrorsfordeterminationofthetilttransformationofeachparticlepair(blackbars)fromV/A-typeATPaseparticleorientationsaremuch

betterthanexpectedforrandomorientations(whitebars).DatafromRef.[69].(d)Tilt-pairalignmenttestshowsthaterrorsforcalculatedtilt

transformationsforbovineATPasesynthase(blackbars)areworsethanthatforV/A-type,butstillbetterthanrandom(whitebars).DatafromRef.

determination, the latter estimated by tilt-pair analysis [23,37].

The Fourier shell correlation (FSC) ([38–40], http://

www.ebi.ac.uk/pdbe/emdb/validation/fsc/) is the most

widely reported global measure of map resolution,

al-thoughothershavebeenproposed[41].TheFSCisthe

correlationbetweentwo3Dmaps,eachcalculatedfrom

an independent half of the data (a ‘half-map’) as a

function of resolution. The FSC is a measure of the

signal-to-noise ratio of the map,which decreases with

increasingresolution.Theresolutionofthemapmaybe

assigned when the FSC drops below a threshold (e.g.

FSC=0.143). However, the FSC curve may show

featuresreflectinganumberofpotentialproblems[42]

andshouldbeplottedfromthelowestresolutionbinto

theNyquistfrequency.Itisalsopossible,duetolimited

accuracy inorientationdeterminationorbecausesome

regionsofthestructurearelesswell-orderedthanothers,

thatthereisnotonesingleresolutionthatapplies

glob-allytothemap.Inthiscase,theFSCcut-offresolutionis

likely to represent the resolution of only part of the

structure,especiallyifalargenumberofparticleimages

havebeenused.

TheFSCcalculationassumesthatthetwohalf-mapsare

independent. Previously, it was deemed sufficient to

separate thedatausedto calculatetheFSCatthefinal

iterationofmapcalculation.However,falseor

exaggerat-edresolutionclaimsmayresultfromthetwohalvesofthe

datasetbeingover-fittothesamespuriousnoisefeatures

inasinglereferencemap.Itisnowacceptedthatunless

othersafeguardsareappliedasdescribedbelow,thetwo

halvesoftheimagedatasetshouldbekeptindependent

Figure2 (a) (d) T Q S P R (c) (b)

Current Opinion in Structural Biology

Mapfeaturesdependonresolution.Atresolutionsworsethanaround20A˚,justtheoverallenvelopeofaproteincomplexwillbeapparent.

a-Helicesareresolvedpast9–10A˚ whiledistinguishingthestrandsofb-sheetsrequiresresolutionsbetterthan4.8A˚.Atapproximately4A˚

resolutionsomeofthebulkysidechainsinaminoacidscanbedetected.Theresolutionatwhichitbecomespossibletobuildanall-atommodel

ofapolypeptideis3.5A˚.Atomicmodelbuildingbecomesincreasinglyreliablewithincreasingresolutionandismostlyrobustaround2.5A˚

resolution.(a)Rotavirusdouble-layeredparticlefilteredto20A˚,(b)filteredto7A˚ showinga-helices,(c)modelbuiltat3.8A˚ resolution.FromRef.

[51].Copyright2008TheNationalAcademyofSciences.(d)RotavirusDLPat2.6A˚ resolution(fromEMDB-6272)[50],imagekindlyprovidedby

during refinement and independent reference maps

should be used [7,9,14].In analogy with the Rfree,

datacouldbeomittedforcross-validationbyleavingouta

bandofspatialfrequencies[43].Morecommonly,

‘reso-lution limited’ refinement is performed, where a

maxi-mum resolution cut-off for thedataused in orientation

determinationduringaniterationofrefinementischosen,

andsignalinthehigherresolution‘freeshells’isdetected

by correlation between the half-maps [9,23,43]. The

resolution limit for each cycle of refinement may be

chosenin an automatic and unbiased wayby usingthe

FSCasaweightonthedataduringorientation

determi-nation[9].In practice, thehighest-resolutiondata

ac-cessible by cryomicroscopy do not contribute to

orientation determination and a lower-resolution limit

may bechosen for particle refinement [23,25].Thus,

resolution-limitedrefinementreducestheover-fittingbut

does not adversely affect the resolution of the final

structuredetermination[9].Inthepast, whenasingle

reference map and all spatial frequencieswere used to

aligndataforthetwohalf-maps,anoverlyoptimisticFSC

curvewasprobablycompensatedbythemore

conserva-tiveFSC=0.5criterionratherthan FSC=0.143[44].

In practice,any refinement, evenwhere fully

indepen-dentlyrefinedmaps are compared, will introducesome

amountofover-fitting.Chenetal. [6]haveproposeda

testtodetectover-fittingbymakinguseofimageswhere

the phases of high-resolution structure factors are

ran-domized or replaced entirely by structure factors from

imagesofnoise.Identicalrefinementsusinganyprotocol

chosen by the investigator are then performed using

theoriginalimagesand thephase-randomizedor

noise-substituted images. The difference between the FSC

from the original data (FSCt) and the FSC with

high-resolution noise substitution or phase randomization

(FSCn) represents the true high-resolution signal in

themap(FSCtrue).(Figure 3a,b).

Molecularmapsdonotentirelyfillthe3Dvolumeofthe

boxinwhichtheyarecalculatedandtheregionoutsideof

themolecularenvelope containsnoise,whichdecreases

theFSCand thereportedresolution of themap.

Typi-callyasoft-edgemaskisappliedtolimittheextentofthe

density.Unfortunately,useofthesamemaskinthetwo

independent maps introduces artificial correlations

be-tweenthemapsleadingtoanover-estimateofresolution,

particularlywhenthemasksaresotightthattheycontact

themapdensity.Forthisreason,authorsshoulddeposit

anunmaskedmapaswellasthemasksappliedtothefinal

volumeusedtocalculatetheFSC[14].High-resolution

noise substitution has also been applied to correct the

FSCfortheeffectofthemask[6](Figure3c).

Despite the value of global resolution measures, it is

increasinglyrecognizedthatcryo-EMmapsdonothave

uniform resolution throughout and possess regions of

weaker density due to disorder and conformational

and/orcompositional heterogeneity. Inaccuracies in the

determination of orientation parameters for particle

imagesleadstotheedgesofmapsbeingatlower

resolu-tionthanthecenterwheretheseinaccuracieshavelessof

aneffect[45].Severalapproacheshavebeenproposed

forexamininglocalresolution[46].TheprogramResMap

searches for the smallest sinusoidal feature that canbe

detectedabovethenoiseateverypointin amap,while

accountingforthefalsediscoveryrateoftheprocess[47].

Methodsfor analysingthevariancein astructure

deter-minationprovideinformationonthemobilityofprotein

regions[48,49].

Validating

maps

with

atomic

coordinate

models

and

building

valid

models

Low-resolution map density may be validated by the

agreementwiththeshapeofanatomicmodeldetermined

byX-raycrystallography,ideallywithquantitative

justifi-cationforauniqueplacementofthemodelasarigidbody

withinthemap[14].Theinterpretationofthemapwith

a model also chooses the absolute hand of the map.

Alternatively,experimentaldeterminationoftheabsolute

handcanbekeyinvalidatingthefitofmodelstomapsat

lowresolution.Wherethestructuresofcomponents

un-dergoconformational changes in thecontext of a large

assembly,theymaybebrokenintoseveralrigidbodiesor

flexiblyfit(ormorphed)intothemapdensity.Inthiscase,

thedensityneedstobeatsufficientlyhighresolutionto

justifytheintroductionofextradegreesoffreedomtothe

model.

Therotavirusdouble-layeredparticleat2.6A˚ resolution

[50,51] (Figure 2) is a spectacular example of the

in-creasingnumberofhigh-resolutionmaps(5mapsbetter

than 3A˚ resolution [50,52–55] and >100 maps with

better than5A˚ resolution inthe Electron Microscopy

Databank (EMDB)) that have made the building and

assessmentofatomicstructuremodelsanareaofintense

activity. Many tools developed for evaluating X-ray

crystallographicmodels[56]maybeappliedincryoEM

[57–59].The crystallographicphase problem resultsin

initialmaps withpoor phases that improve during the

courseofrefinementsothat modelbuildingisan

inte-gralpartofhigh-resolutionmapdetermination.In

cryo-EM, the refinement of the experimental map is

pres-entlyaseparatestepbeforemodelbuilding.Anatomic

modelmaythenberefinedusingatarget thatincludes

mapamplitudesandphasesaswellastermsformodel

bondingandstereochemistry.Amodelmayalsobebuilt

forpart of amap whose boundary is definedby asoft

mask[60].EMmapsaretypically atlowerresolution

thanX-ray diffraction data andnew tools for building

andrefiningintotheselowerresolutionmaps[60,61–

63] incorporate prior information or additional

con-straintsthat reducethe degreesoffreedom and

As withX-ray models, measuresof agreement between

map and modelinclude global statistics aswell aslocal

measuresatspecificmainchainandaminoacidside-chain

positions. The FSC calculated betweenthe model and

thefullmap(Cref)isaglobalmeasureoftheresolutionof

amap.Foraperfectmodelandamapthatdoesnothave

variable resolutionin differentregionsthe resolutionat

which Crefcrosses0.5shouldbein agreementwiththe

resolutionassignedusingthe0.143thresholdintheFSC

between experimental half-maps [23]. With maps that

haveavariableresolutionhowever,themap/modelCref

plot mayhave Cref=0.5at alowerresolution than the

half-map resolution at FSC=0.143, implying that the

model does not explain the map density quite as well

asexpected.Thedisagreementmaybehidden

(purpose-fullyorunintentionally)byflexiblefittingthatallowstoo

manydegreesoffreedomleadingtoanoverlyoptimistic

CrefasinFigure3d[6].Aswithmaprefinement,

high-resolution databeyond acut-off maybeexcluded from

model refinement with the high-resolution data range

scored as free shells [64]. The independently refined

half-maps used to calculate the FSC have also been

exploited for separate model refinementand

cross-vali-dation [60,65]. For example, in refinement of the

Figure3 0.0 0.5 1.0 FSCn unmasked spherical mask soft shaped mask steep shaped mask

0.00 0.05 0.10 0.15 0.20 0.0 0.5 1.0 Resolution(1/Å) 0.00 0.05 0.10 0.15 0.20 Resolution(1/Å) FSC (map.vs.model)

FSC Relion map vs 3i3e FSC (vs Jelly body)

(c) (d) (a) (b) 0.0 0.5 1.0 FSC 0.00 0.05 0.10

film data refined to 7Å

17Å 10Å 17Å 10Å

17Å HR-noise refined to 7Å 17Å HR-rand refined to 7Å

film data refined to 17Å 17Å HR-noise refined to 17Å 17Å HR-rand refined to 17Å 0.15 0.20 Resolution (1/Å) 0.0 0.5 1.0 FSC 0.00 0.05 0.10 0.15 0.20 Resolution (1/Å)

Current Opinion in Structural Biology

ResolutionassessmentbyFouriershellcorrelation(FSC)anditsvalidationbythenoisesubstitutiontest.(a)FSCbetweenhalf-mapswhere

particleorientationrefinementusesimagedatato7A˚ resolution(redcircle)orto7A˚ butwherehighresolutiondatabeyond17A˚ hasbeen

substitutedwithdatawithrandomizedphases(HR-noise,greencircle)ordatafromnoiseimages(HR-noise,bluecircle).PortionofFSCsignaldue

toover-fittingisshownbytheshadedblueregion(FSCn),andtheportionduetotruestructuralsignalistheshadedpinkregion(FSCtrue).(b)

FSCcurvesasabovebutwhereparticleorientationrefinementhasbeenlimitedto17A˚ resolution.The‘freeshells’above17A˚ resolutionshowno

over-fittingbecausetheywerenotusedinrefinement.(c)FSCnbetweenhalf-mapsfornoise-substituteddatashowsFSCartefactsthatresult

fromatightmask.(d)FSCcurve(red),alsocalledCref,betweenwhole-mapandcoordinatemodelplacedinthemapbyrigidbodyfittingcrosses

the0.5thresholdatapointindicating7.6A˚ resolution.FlexiblefittingofthemodelproducesanoverlyoptimisticFSCcurve(blue).Panelsare

model for the 3.3A˚ mitochondrial ribosome, the

FSCwork between model and one half-map measured

in‘freeshells’athighresolutionwascomparedwiththe

FSCtest in the second half-map in ‘free shells’ of the

sameresolutionrange[60,66,67].

Conclusions

and

remaining

problems

Structure determination bycryo-EM of single particles

will continueto improve andwill be extendedto more

challenging lower molecular weight and heterogeneous

specimens. New, robust validation tools will need to

accompanythisdevelopment.High-resolutionmap

fea-turessuchashelicesandside-chainsarestrongevidence

ofavalidstructuredetermination. However,becauseof

the danger of model bias, it must be clear that these

features were not introduced from an external source

duringparticlepicking,2Dalignment,or3Dalignment,

suchasbyaligningparticleimageswithanotherEMmap

or map derived from acoordinatemodel. Wetherefore

recommend performing as many tests as possible to

communicatethevalidityofamap.Furthermore,

valida-tiontoolsmaybeappliedtooptimizingmapcalculationor

fortestingnewcomputationalprocedures.Forexample,

tilt-pair analysis tests the validity of a model at

low-resolution but also demonstrates accurate orientation

determination, which is a condition for high-resolution

structuredeterminationfrommodestnumbersofparticle

images.

Heterogeneous particles raise several new problems in

cryo-EMforwhichnewvalidationtoolsmayberequired.

High-resolution maps may be calculated from a small

subset of the data (e.g. 5%) [68]. From a validation

standpoint,suchamapmaybecorrectbutnotrepresent

theentirepopulationofparticles.Procedurestovalidate

small differences in maps at high resolution will be an

importantdevelopment.Aswithmacromolecular

crystal-lography,publicrepositories suchas theEMDB (www.

emdatabank.org)willplayanincreasinglyimportantrole

inprovidingstandardizedvalidationtoolsandvalidation

reportsthatwillbeavailablewhiledownloadingmapsand

models. Computational tools, however, do not replace

rigor at the biochemistry, specimen preparation, and

image acquisition stages of the experiment: there will

always be circumstances under which even the most

robusttoolscanbemisled.

Conflict

of

interest

statement

Nothingdeclared.

Acknowledgements

P.B.R.issupportedbytheFrancisCrickInstitutewhichreceivesitscore

fundingfromCancerResearchUK,theUKMedicalResearchCouncil

(programcodeU117581334),andtheWellcomeTrust.J.L.R.issupported

byoperatinggrantMOP81294fromtheCanadianInstitutesofHealth

Research,discoverygrant401724-12fromtheNaturalSciencesand

EngineeringResearchCouncil,andtheCanadaResearchChairsprogram.

WethankRichardHendersonforcommentsonthemanuscript.

References

and

recommended

reading

Papersofparticularinterest,publishedwithintheperiodofreview,

havebeenhighlightedas:

 ofspecialinterest

ofoutstandinginterest

1. HendersonR:Thepotentialandlimitationsofneutrons, electronsandX-raysforatomic-resolutionmicroscopyof unstainedbiologicalmolecules.QuartRevBiophys1995, 28:171-193.

2. 

HendersonR:Avoidingthepitfallsofsingleparticle cryo-electronmicroscopy:Einsteinfromnoise.ProcNatlAcadSciU SA2013,110:18037-18041.

Perspectiveonproblemsinsingleparticlemicroscopyrelatedtomodel

bias,andrecommendationsonstrategiesandvalidationtoolstoavoid

incorrectstructuresorexaggeratedresolutionclaims.

3. PenczekP,RadermacherM,FrankJ:Three-dimensional reconstructionofsingleparticlesembeddedinice. Ultramicroscopy1992,40:33-53.

4. SigworthFJ:Amaximum-likelihoodapproachto single-particleimagerefinement.JStructBiol1998,122:328-339.

5. vanHeelM:FindingtrimericHIV-1envelopeglycoproteinsin randomnoise.ProcNatlAcadSciUSA2013,110:E4175-E4177.

6. 

ChenS,McMullanG,FaruqiAR,MurshudovGN,ShortJM, ScheresSH,HendersonR:High-resolutionnoisesubstitutionto measureoverfittingandvalidateresolutionin3Dstructure determinationbysingleparticleelectroncryomicroscopy. Ultramicroscopy2013,135:24-35.

TesttodemonstratewhetherthecalculatedFouriershellcorrelation(FSC)

isanaccurateassessmentofstructuralsignal athigh resolutionand

determinesportionofFSCsignalduetoover-fitting.

7. GrigorieffN:Resolutionmeasurementinstructuresderived fromsingleparticles.ActaCrystallogrDBiolCrystallogr2000, 56:1270-1277.

8. StewartA,GrigorieffN:Noisebiasintherefinementof structuresderivedfromsingleparticles.Ultramicroscopy2004, 102:67-84.

9. 

ScheresSH,ChenS:Preventionofoverfittingincryo-EM structuredetermination.NatMethods2012,9:853-854.

DemonstrationthatusingFSCweightingtoperformfrequencylimited

refinementofindependenthalf-mapsreducesover-fittingwithout

redu-cingalignmentaccuracyandfinalmapresolution.

10. BrungerAT:FreeRvalue:anovelstatisticalquantityfor assessingtheaccuracyofcrystalstructures.Nature1992, 355:472-475.

11. LyumkisD,BrilotAF,TheobaldDL,GrigorieffN: Likelihood-basedclassificationofcryo-EMimagesusingFREALIGN.J StructBiol2013,183:377-388.

12. ScheresSH:RELION:implementationofaBayesianapproach tocryo-EMstructuredetermination.JStructBiol2012, 180:519-530.

13. ScheresSH,CarazoJM:Introducingrobustnessto maximum-likelihoodrefinementofelectron-microscopydata.Acta CrystallogrDBiolCrystallogr2009,65:672-678.

14. 

HendersonR,SaliA,BakerML,CarragherB,DevkotaB, DowningKH,EgelmanEH,FengZ,FrankJ,GrigorieffNetal.: Outcomeofthefirstelectronmicroscopyvalidationtaskforce meeting.Structure2012,20:205-214.

ReviewoftheinauguralmeetingoftheElectronMicroscopyValidation

TaskForceorganizedbytheElectronMicroscopyDatabank

(www.em-databank.org).Reportsconsensusonmapandmodel validationand

databasedeposition.

15. ShaikhTR,TrujilloR,LeBarronJS,BaxterWT,FrankJ: Particle-verificationforsingle-particle,reference-based

reconstructionusingmultivariatedataanalysisand classification.JStructBiol2008,164:41-48.

16. YangZ,FangJ,ChittuluruJ,AsturiasFJ,PenczekPA:Iterative stablealignmentandclusteringof2Dtransmissionelectron microscopeimages.Structure2012,20:237-247.

17. SchatzM,vanHeelM:Invariantclassificationofmolecularviews inelectronmicrographs.Ultramicroscopy1990,32:255-264.

18. RubinsteinJL,WalkerJE,HendersonR:Structureofthe mitochondrialATPsynthasebyelectroncryomicroscopy. EMBOJ2003,22:6182-6192.

19. BakerLA,RubinsteinJL:Angledeterminationforsideviewsin singleparticleelectronmicroscopy.JStructBiol2008, 162:260-270.

20. SorzanoCO,MarabiniR,BoissetN,RietzelE,SchroderR, HermanGT,CarazoJM:Theeffectofoverabundantprojection directionson3Dreconstructionalgorithms.JStructBiol2001, 133:108-118.

21. CrowtherRA,AmosLA:Harmonicanalysisofelectron microscopeimageswithrotationalsymmetry.JMolBiol1971, 60:123-130.

22. DubeP,TavaresP,LurzR,vanHM:Theportalproteinof bacteriophageSPP1:aDNApumpwith13-foldsymmetry. EMBOJ1993,12:1303-1309.

23. RosenthalPB,HendersonR:Optimaldeterminationofparticle orientation,absolutehand,andcontrastlossinsingle-particle electroncryomicroscopy.JMolBiol2003,333:721-745.

24. 

WasilewskiS,RosenthalPB:Webserverfortilt-pairvalidationof singleparticlemapsfromelectroncryomicroscopy.JStruct Biol2014,186:122-131.

Web-based tilt-pair server nowavailable at European Bioinformatics

Institute (http://www.ebi.ac.uk/pdbe/emdb/validation/tiltpair/). Paper

describesadditionalexamplesandproceduresforusingtilt-pairresidual

plots. 25. 

HendersonR,ChenS,ChenJZ,GrigorieffN,PassmoreLA, CiccarelliL,RubinsteinJL,CrowtherRA,StewartPL, RosenthalPB:Tilt-pairanalysisofimagesfromarangeof differentspecimensinsingle-particleelectron

cryomicroscopy.JMolBiol2011,413:1028-1046.

Applies tilt-pairparameter plotstoarange ofspecimens ofdifferent

molecular weight, shows the difference inorientation accuracyas a

function ofmolecularweight, andrelates these tothecomputational

temperaturefactordescribingcontrastloss.

26. 

BakerLA,WattIN,RunswickMJ,WalkerJE,RubinsteinJL: Arrangementofsubunitsinintactmammalianmitochondrial ATPsynthasedeterminedbycryo-EM.ProcNatlAcadSciUSA 2012,109:11675-11680.

Methods include quantificationof accuracy in tilt pair tests through

comparisonoferrordistributionswiththatforrandomorientations.

27. 

RussoCJ,PassmoreLA:Robustevaluationof3Delectron cryomicroscopydatausingtilt-pairs.JStructBiol2014, 187:112-118.

Statisticalanalysisoftilt-pairparameterplotsdescribingaconcentration

parameterkontheFischerdistribution.

28. SchreiberA,StengelF,ZhangZ,EnchevRI,KongEH,MorrisEP, RobinsonCV,daFonsecaPC,BarfordD:Structuralbasisforthe subunitassemblyoftheanaphase-promotingcomplex.Nature 2011,470:227-232.

29. ChenZ,SpeckC,WendelP,TangC,StillmanB,LiH:The architectureoftheDNAreplicationoriginrecognitioncomplex inSaccharomycescerevisiae.ProcNatlAcadSciUSA2008, 105:10326-10331.

30. KimJ,WuS,TomasiakTM,MergelC,WinterMB,StillerSB, Robles-ColmanaresY,StroudRM,TampeR,CraikCSetal.: Subnanometre-resolutionelectroncryomicroscopystructure ofaheterodimericABCexporter.Nature2015,517:396-400.

31. SmithTJ,OlsonNH,ChengRH,LiuH,ChaseES,LeeWM, LeippeDM,MosserAG,RueckertRR,BakerTS:Structureof humanrhinoviruscomplexedwithFabfragmentsfroma neutralizingantibody.JVirol1993,67:1148-1158.

32. VinothkumarKR,McMullanG,HendersonR:Molecular mechanismofantibody-mediatedactivationof b-galactosidase.Structure2014,22:621-627.

33. RubinsteinJ,WalkerJ:ATPsynthasefromSaccharomyces cerevisiae:locationoftheOSCPsubunitintheperipheralstalk region.JMolBiol2002,321:613-619.

34. Al-BassamJ,OzerRS,SaferD,HalpainS,MilliganRA:MAP2and taubindlongitudinallyalongtheouterridgesofmicrotubule protofilaments.JCellBiol2002,157:1187-1196.

35. BottcherB,WynneSA,CrowtherRA:Determinationofthefoldof thecoreproteinofhepatitisBvirusbyelectron

cryomicroscopy.Nature1997,386:88-91.

36. FernandezJJ,LuqueD,CastonJR,CarrascosaJL:Sharpening highresolutioninformationinsingleparticleelectron cryomicroscopy.JStructBiol2008,164:170-175.

37. StaggSM,NobleAJ,SpilmanM,ChapmanMS:ResLogplotsas anempiricalmetricofthequalityofcryo-EMreconstructions. JStructBiol2014,185:418-426.

38. SaxtonWO,BaumeisterW:Thecorrelationaveragingofa regularlyarrangedbacterialcellenvelopeprotein.JMicrosc 1982,127:127-138.

39. vanHeelM,HarauzG:Resolutioncriteriaforthreedimensional reconstruction.Optik1986,73:119-122.

40. vanHeelM,Stoffler-MeilickeM:CharacteristicviewsofE.coli andB.stearothermophilus30Sribosomalsubunitsinthe electronmicroscope.EMBOJ1985,4:2389-2395.

41. SousaD,GrigorieffN:Abinitioresolutionmeasurementfor singleparticlestructures.JStructBiol2007,157:201-210.

42. PenczekPA:Resolutionmeasuresinmolecularelectron microscopy.MethodsEnzymol2010,482:73-100.

43. ShaikhTR,HegerlR,FrankJ:Anapproachtoexaminingmodel dependenceinEMreconstructionsusingcross-validation. JStructBiol2003,142:301-310.

44. GrigorieffN,HarrisonSC:Near-atomicresolution

reconstructionsoficosahedralvirusesfromelectron cryo-microscopy.CurrOpinStructBiol2011,21:265-273.

45. 

KishchenkoGP,LeithA:Sphericaldeconvolutionimproves qualityofsingleparticlereconstruction.JStructBiol2014, 187:84-92.

Reportscomputationalsharpeningprocedurereducingblurathighradius

associatedwithparticlemis-orientation

46. CardoneG,HeymannJB,StevenAC:Onenumberdoesnotfit all:mappinglocalvariationsinresolutionincryo-EM reconstructions.JStructBiol2013,184:226-236.

47. KucukelbirA,SigworthFJ,TagareHD:Quantifyingthe localresolutionofcryo-EMdensitymaps.NatMethods2014, 11:63-65.

48. SpahnCM,PenczekPA:Exploringconformationalmodesof macromolecularassembliesbymultiparticlecryo-EM.Curr OpinStructBiol2009,19:623-631.

49. ZhangW,KimmelM,SpahnCM,PenczekPA:Heterogeneityof largemacromolecularcomplexesrevealedby3Dcryo-EM varianceanalysis.Structure2008,16:1770-1776.

50. GrantT,GrigorieffN:Measuringtheoptimalexposureforsingle particlecryo-EMusinga2.6A˚ reconstructionofrotavirusVP6. Elife2015:4.

51. ZhangX,SettembreE,XuC,DormitzerPR,BellamyR, HarrisonSC,GrigorieffN:Near-atomicresolutionusingelectron cryomicroscopyandsingle-particlereconstruction.ProcNatl AcadSciUSA2008,105:1867-1872.

52. BartesaghiA,MerkA,BanerjeeS,MatthiesD,WuX,MilneJL, SubramaniamS:2.2A˚ resolutioncryo-EMstructureof beta-galactosidaseincomplexwithacell-permeantinhibitor. Science2015,348:1147-1151.

53. CampbellMG,VeeslerD,ChengA,PotterCS,CarragherB:2.8A˚ resolutionreconstructionoftheThermoplasmaacidophilum 20Sproteasomeusingcryo-electronmicroscopy.Elife 2015:4.

54. FischerN,NeumannP,KonevegaAL,BockLV,FicnerR, RodninaMV,StarkH:StructureoftheE.coliribosome-EF-Tu complexat<3A˚ resolutionbyCs-correctedcryo-EM.Nature 2015,520:567-570.

55. JiangJ,PenteluteBL,CollierRJ,ZhouZH:Atomicstructureof anthraxprotectiveantigenporeelucidatestoxin

translocation.Nature2015,521:545-549.

56. ReadRJ,AdamsPD,Arendall WB3rd,BrungerAT,EmsleyP, JoostenRP,KleywegtGJ,KrissinelEB,LuttekeT,OtwinowskiZ etal.:Anewgenerationofcrystallographicvalidationtoolsfor theproteindatabank.Structure2011,19:1395-1412.

57. EmsleyP,CowtanK:Coot:model-buildingtoolsformolecular graphics.ActaCrystallogrDBiolCrystallogr2004,60:2126-2132.

58. MurshudovGN,SkubakP,LebedevAA,PannuNS,SteinerRA, NichollsRA,WinnMD,LongF,VaginAA:REFMAC5forthe refinementofmacromolecularcrystalstructures.Acta CrystallogrDBiolCrystallogr2011,67:355-367.

59. WinnMD,BallardCC,CowtanKD,DodsonEJ,EmsleyP, EvansPR,KeeganRM,KrissinelEB,LeslieAG,McCoyAetal.: OverviewoftheCCP4suiteandcurrentdevelopments.Acta CrystallogrDBiolCrystallogr2011,67:235-242.

60. 

BrownA,LongF,NichollsRA,TootsJ,EmsleyP,MurshudovG: Toolsformacromolecularmodelbuildingandrefinementinto electroncryo-microscopyreconstructions.ActaCrystallogrD BiolCrystallogr2015,71:136-153.

Comprehensivediscussionofmodelbuildingandrefinementinto

cryo-EMmapsincludingcross-validation.UsesFSCworkinone-halfmapand

FSCtestinsecondhalf-mapwithafurtherrestrictiontofree-shellsathigh

resolution.

61. DiMaioF,TykaMD,BakerML,ChiuW,BakerD:Refinementof proteinstructuresintolow-resolutiondensitymapsusing rosetta.JMolBiol2009,392:181-190.

62. HeaddJJ,EcholsN,AfoninePV,Grosse-KunstleveRW,ChenVB, MoriartyNW,RichardsonDC,RichardsonJS,AdamsPD:Useof knowledge-basedrestraintsinphenix.refinetoimprove

macromolecularrefinementatlowresolution.ActaCrystallogr DBiolCrystallogr2012,68:381-390.

63. NichollsRA,LongF,MurshudovGN:Low-resolutionrefinement toolsinREFMAC5.ActaCrystallogrDBiolCrystallogr2012, 68:404-417.

64. 

FalknerB,SchroderGF:Cross-validationincryo-EM-based structuralmodeling.ProcNatlAcadSciUSA2013, 110:8930-8935NatlAcadSciUSA,110,2013,8930–8935.

Cross-validationofatomicmodelrefinementemployingfreeshellsathigh

resolution. 65. 

DiMaioF,ZhangJ,ChiuW,BakerD:Cryo-EMmodelvalidation usingindependentmapreconstructions.ProteinSci2013, 22:865-868.

Cross-validationofmodelrefinementbybuildingacoordinatemodelinto

onehalf-mapandmonitoringtheFouriershellcorrelation(FSC)onsecond

half-map.

66. AmuntsA,BrownA,BaiXC,LlacerJL,HussainT,EmsleyP, LongF,MurshudovG,ScheresSH,RamakrishnanV:Structureof theyeastmitochondriallargeribosomalsubunit.Science2014, 343:1485-1489.

67. BrownA,AmuntsA,BaiXC,SugimotoY,EdwardsPC, MurshudovG,ScheresSH,RamakrishnanV:Structureofthe largeribosomalsubunitfromhumanmitochondria.Science 2014,346:718-722.

68. FernandezIS,BaiXC,HussainT,KelleyAC,LorschJR, RamakrishnanV,ScheresSH:Moleculararchitectureofa eukaryotictranslationalinitiationcomplex.Science2013, 342:1240585.

69. LauWC,RubinsteinJL:StructureofintactThermus thermophilusV-ATPasebycryo-EMrevealsorganizationof themembrane-boundV(O)motor.ProcNatlAcadSciUSA 2010,107:1367-1372.