Altmetrics
in
Humanities
and
Social
Sciences
KathleenFitzpatrickandRebeccaKennison 30October2017
Introduction
Thespreadofopendigitalformsofscholarlycommunication,combinedwithincreasing
institutionalpressuretotrackresearch“impact,”hasencouragedscholarsandadministratorsin thehumanitiesandsocialsciencestoturntheirattention tometricsthatpromise tohelpinthe assessmentofresearchoutputs.However,significantconcernshavebeenraisedinrecent yearsaboutthevalueoftraditionalmetricsin suchassessment.Forinstance,thejournalimpact factor—asitsnamewouldsuggest—onlymeasurestheimpactofa publicationasawhole, notthesignificanceofanyindividualpieceof workthatitcontains.Similarly,citationmetrics suchastheh-index,whileauthorspecific,onlyreveal asingleaspectof theimpactascholar’s workmayhave,failingtoaccountforthewaysthatanarticlemovesthroughdigitalscholarly networkstoday.Moreover,citationmetrics’focus onjournal-basedcitationsmakethem particularlyinapplicableinthefieldswithinthehumanities andsocialsciencesthatdonotrely onpeer-reviewedarticlesastheprimaryform ofscholarlycommunication. (Ontheshortcomings ofandpotentialdamagedonebytraditionalbibliometricsintheassessment ofscholarship,see Burrows, 2016; de Rijcke and Rushforth, 2015; Gruber, 2014; Haustein and Larivière, 2015. On thespecificproblemswithusingsuchbiblometrics inhumanitiesand socialsciences,see Archambaultetal.,2006;Nederhof,2006;Nederhofetal.,1989;Pontille andTorny,2010.) Asaresultofthelimitationsoftraditional bibliometrics,anumberofalternativemetricssystems formeasuringresearchimpacthaverecently gainedpopularity,especiallyinscience,
technology,engineering,andmedicine,knowncollectively astheSTEMfields(Hammarfelt, 2014;Hug,Ochsner,andDaniel,2013;Koushaand Thelwall,2016;Priemetal.,2010;Thelwall andDelgado,2015).Theseso-called“altmetrics”attempttoaccountnotmerely forcitationsof publishedscholarshipinjournal-basedarticles,butalsomentionsofthework inpopularnews outlets,inboundlinkstotheworkfromsocialmediasuchasTwitterandFacebook,andcapture oftheworkinsocialbookmarkingandcitationmanagementsystemssuch asMendeleyand Zotero,andseektotrackotherfactorsthat collectivelyindicatethewaysthatapublication movesacrosstheInternet.Whileskepticsarguethatsocialmediaattention doesnotequal quality,relevance,orimpact(Bornmann,2014;Scott, 2012),andwhile graveconcernsexist regardingthepotentialusesandabusesofmetricsinpersonnelreviews (Flaherty,2016;Laudel and Gläser, 2006), promoters of such alternative metrics suggest that they provide new insights intothewaysthatscholarlyworkisdisseminatedbyitscreatorsandusedbyitsaudiences. Toassessthecurrentstateofaltmetricswithinhumanitiesandsocial sciencesdisciplines,this studyproposedtodevelopataxonomyofthe altmetricstoolsandmeasuresmostwidelyused byorfamiliartoresearchersandscholars,withthegoalofdeterminingthecurrentlevelof acceptancewithintheacademiccommunityofaltmetrics,especiallyinrelationtodecisions
concerningtenureandpromotion.Oursense,inbeginning thisstudy,wasthatwewouldmeet withafairdegreeofconcernabouttheeffects ofapplyingmetricsdeveloped forthesciencesto fieldsthatoperatewithquitedifferentstructuresthroughwhichworkcirculates. Ourhopewas thatwemightprovidesomeguidancefordepartment chairsanddeansinhumanitiesandsocial sciencefieldsastheyencounterrequestsfor analyticdataat theuniversitylevel.
Traditionally,awhitepapersuchasthisonewouldbeginwithathoroughliteraturereview.We weredissuadedfromdoingsointhiscasebytheexperienceoftheHigherEducationFunding CouncilforEngland(HEFCE),whichin2014–2015 commissionedareview ofthecurrent landscapeformetricsinresearchevaluation.Aspart ofthatreview,ateamofresearchers primarilyaffiliatedwiththeCentreforScience andTechnologyStudies atLeidenUniversity (SarahdeRijcke,PaulF.Wouters,AlexD.Rushforth,ThomasP. Franssen,andBjörn
Hammarfelt)conductedananalysisoftheextantliteratureonevaluationpractices andtheuses andmisusesofmetricstherein.Theynotedin theprocessthedifficultiesinvolvedinaccounting forthefullbreadth oftheliterature:
Providingacompleteoverviewoftheliteratureis notfeasibleforacoupleofreasons. Firstofall,theliteratureisvery diverse.Studieson evaluationsystems,evaluation practices,andeffectsofindicatorusesare publishedindifferent media,andthe
preferredoutletsarenotnecessarilyalwaysinternationaljournalsthatarecoveredwell byweb-basedcitationdatabases.Thehundredsof sourcesarespread outoverbooks, editedvolumes,articles,reports,andotherforms ofgrayliterature thataresometimes relativelyinaccessible.Secondly,therelevantliteratureisscatteredoveralargenumber ofsocialsciencefields,includingsociologyofscience,innovationstudies,libraryand informationscience,highereducationstudies, sociologyofevaluation,evaluation studies,economicsandbusinessstudies,medicalsociology,sciencepolicystudies, researchmanagementandinnovation,politicalscience, andgovernancestudies.Athird hamperingfactorinpresentingacompleteoverview istheepistemic natureofthe evidencepresentedintheliterature.Thestudies rangefromsurveys andinterviewson researchers’perceptionsofevaluationsandformalpolicyanalysisofprincipal–agent relationshipstoculturalcritiquesoftheevaluationsocietyandethnographicstudiesof evaluationinaction(anemergingbodyof work).Theresulting heterogeneityofthe evidenceposesparticularchallengesinintegratingthe literatureina singlereview.(de Rijckeetal.,2016,161–162)
DeRijckeetal.optintheirreporttopresentaqualitativeliteraturereview thatmapstheprimary issuesacrosstheliteratureratherthanstrivingforcompleteness.Needlessto say,sincetheir reportwaspublished,thequantityanddiversityof theliteraturehaveonlyexpanded,andour ownreadingoftherelevantworksupportedthesoundnessoftheir decision.Moreover,the primarythemesthattheirreviewuncovered—questionsabouttheeffectsoftheuseofmetrics onknowledgeproductionandtheconsequencesfortheirdeploymentinresearchassessment —wereconfirmedbothfromourreadingandfrom ourprimaryresearch.As aresult,wehave optedtorefertheinterestedreadertotheir study,ratherthanreinventthatparticularwheel.
AsdeRijckeetal.note,muchoftheextantliteratureconnectstherise ofnewmetricsfor researchimpact,includingso-called“altmetrics,”toanincreaseindemandsforresearcher accountability(see,e.g.,studiesoftheimpactoftheResearchAssessmentExercise/Research ExcellenceFrameworkonresearchers,includingCollini, 2012;Hoecht,2006).Thedeployment ofmetricsinresearchassessmentinmany casessteersresearchers tobecomemore
market-oriented,moreinstrumentalist,andmoreprivatized (seeLeisyteandDee,2012; Willmott,2011).Moreover,assessmentsystemsthat affectresearchers’fundingorreputations willtendtocausethemtoshifttheirgoalstofocusontheoutcomes oftheassessment,rather thanthepurposesoftheresearch,orwill otherwiseencouragethemtorevisetheirprocesses soastoavoidrisk(seeHicks,2012).De Rijckeetal.express particularconcernaboutthese effectsintheareasofthecurriculumonwhich ourownprojectmost focuses;inparticular,they citestudiesindicatingthattheuseof metricsinresearch assessmentcanhinder
interdisciplinaryresearch.Theyalsonotetheextenttowhichtheartsandhumanities,aswellas thebook-orientedsocialsciences,sufferwhenscholars inthesedisciplines areevaluatedbased onmetricsthathavebeendesignedforfields whoseresearchoutputsareentirelycenteredin journals.
Theseconcernswereaprimarydriverbehindthestudythatfollows.Wesoughtamoredirect understandingofthestateofaltmetricsadoptionand usageintheevaluationofresearchin humanitiesandsocialsciencefields,aswell anunderstandingof facultyandadministrator perceptionsofthatusage.Whereconcernsaboutthe usesofmetricsinthehumanitiesand socialsciencesremain,wealsosoughttobeginanexplorationof waysscholarsand
administratorsinthefieldsweaddressmightseektoprovidebetterformsofarticulationofthe desiredimpactofresearch.
Methods
Phase1ofthisstudyincludedtheabovereview oftheliteratureand oftargetedsocialmedia outlets.Whilethereisarelativelysignificantbody ofliteraturerelatedtoaltmetricsinthe sciences,therearefewerstudiestodatelooking atdisciplinesinthehumanitiesandsocial sciences.Tohelpfillthislacuna,thisphaseofthestudyfocusedonreviewing theworkthatis currentlybeingdoneonaltmetricsacrossthedisciplines, aswellasongatheringdiscussionsof metricsinthehumanitiesandsocialsciences bysamplingsocial mediaoutlets,including academicblogsandTwitterfeeds.Wealsoexploreddiscussionsofconcerns aboutmetricsin thehighereducationmainstreampressandin professionalpublications,and weconducteda smallfocusgroupwithhumanitiesandsocialsciencedeansdesignedto surfacethequestions andconcernstheyhaveaboutmetricsandtheiruses.The aimofthisphaseofthestudywasto identify,summarize,andsynthesizethecurrentstate ofaltmetricswithin theacademy;to developaninitialtaxonomyofthetypesof metricsmostcommonlyusedorknown,whether withinSTEM,humanities,orsocialsciences;andtoderivefromthis investigationthequestions tobeexploredinphases2and3.
Phase2ofthestudywasoriginallytoconsistofin-depthinterviewswithapproximately10–12 tenuredandtenure-trackfacultymembersandacademicadministratorsfromarepresentative sampleofNorthAmericaninstitutions,includingliberal artscollegesand midsizedandlarge publicandprivateuniversities,evenlydistributed betweenhumanitiesandsocialscience disciplines.Theseinterviews,conductedviaonlinequestionnaire, weretobe usedtovalidate thetaxonomyandfurtherexploretheissuesanddiscussionstakingplace withindifferenttypes ofinstitutionsanddifferentdisciplinessurroundingtheadoptionofaltmetrics.Resultsand analysisofthequestionnaireweretobeusedtodevelopthesurveyinstrumentforthethird phaseofthestudy.However,ourattemptsto conducttheseinterviewswerechallengedbya surprisinglyhighlackofresponseamongthe initiallyselectedparticipants. Weareuncertain whetherthissilencehadtodowiththebusyness oftheparticipantsor theirperceptionthatthe subjectdidnotpertaintothem.Afterseveralfollow-ups andintheinterest ofmovingthestudy forward,wemadethedecisioninsteadtoopenupthe survey,aspartofphase3,inorderto capturetheresponsesofasmanyinterestedscholarsaspossible.
Phase3involvedaninternationalonlinesurveythat soughttoprovideuswithabroader perspectiveontheuseandlevelofacceptance ofaltmetricsinthehumanitiesandsocial
scienceincasesoftenureandpromotion.We alsosoughtparticipationfromrespondentsfroma rangeofkindsofinstitutionsandfromas wideageographicaldistributionaspossible.We invitedparticipationbyreachingouttohumanitiesandsocialsciencelistservs andtosocial medianetworkstomaximizeourresponserate.Wethenusedresultsfromthissurveytorefine thetaxonomyofmetricsandtoprovidefurtherdataontheissues,use,andacceptanceof altmetricswithinhighereducationinthehumanities andsocialsciences withinaprimarily English-speakingeducationalenvironment.
Preliminary
Interview
Webeganourinvestigationofthewaysthatmetricsingeneral,andaltmetricsinparticular,are beingusedintheevaluationofresearchproductivity inthehumanitiesandsocialsciencesby conductingaone-hourinterviewwithadeanofsocialsciencesatanelitelargeprivateresearch university(“LargePrivateU”).Weexpectedto heararecognizable storyaboutthe
assessment-basedpressurestowardthequantitative engenderedbycontemporaryuniversity bureaucracies.Instead,thisdeanquicklyreframedour work:thisparticular institutionnotonly resistsmodesofassessmentthatareseenasbeingoverly“bean-counting,” buthasavoided developinginternalmetricsforassessingperformanceata rangeoflevels.Thedeannotedthat atLargePrivateUevenbasicdataaremissing andthatwhatdata existaresiloedand
inconsistentlycaptured.Annualreviews,forinstance,rely onnarrativeself-reports sentfirstto thechair,andthentothedean,intheformofaWorddocument. LargePrivateU’scourse managementsystemcannotgeneratereportscontainingusabledata,andthedean’sfinancial reportsarelimitedtoendowmentandgiftaccountsandfacultyresearchaccounts.
Whenweturnedtheconversationspecificallytothe useofmetricsoranalyticsinpersonnel processessuchastenureandpromotionreviews,thedeanletusknowthatLargePrivateU
