Evaluating research: A multidisciplinary approach to assessing research practice and quality

(1)

ContentslistsavailableatScienceDirect

Research

Policy

j ou rn a l h om ep a g e :w w w . e l s e v i e r . c o m / l o c a t e / r e s p o l

Evaluating

research:

A

multidisciplinary

approach

to

assessing

research

practice

and

quality

Pär

Mårtensson

a,1

_,

_Uno

_Fors

b,2

_,

_Sven-Bertil

_Wallin

c,3

_,

_Udo

_Zander

d,4

_,

Gunnar

H

Nilsson

e,∗

a_Department_of_Management_and_{Organization,}_Stockholm_School_of_Economics,_Sveavägen_65,_SE-113₈₃_Stockholm,_Sweden b_Department_of_Computer_and_Systems_Sciences,_Stockholm_University,_PO_Box_7003,_SE-164₀₇_Kista,_Sweden

c_Almain_AB,_Lilla_Norrby,_SE-178₉₁_Munsö,_Sweden

d_Department_of_Marketing_and_Strategy,_Stockholm_School_of_Economics,_Sveavägen_65,_SE-113₈₃_Stockholm,_Sweden e_Department_of_{Neurobiology,}_Care_Sciences_and_Society,_Karolinska_Institutet,_SE-171₇₇_Stockholm,_Sweden

a

r

t

i

c

l

e

i

n

f

o

Articlehistory: Received17June2014 Receivedinrevisedform 23November2015 Accepted24November2015 Availableonline24December2015 Keywords: Researchpractice Researchquality Researchevaluation Multidisciplinary Conceptmodel Rigour Relevance

a

b

s

t

r

a

c

t

Therearefewwidelyacknowledgedqualitystandardsforresearchpractice,andfewdefinitionsofwhat constitutesgoodresearch.Theoverallaimwasthereforetodescribewhatconstitutesresearch,andthen tousethisdescriptiontodevelopamodelofresearchpracticeandtodefineconceptsrelatedtoitsquality. Theprimaryobjectivewastoexploresuchamodelandtocreateamultidisciplinaryunderstandingofthe genericdimensionsofthequalityofresearchpractice.Eightconceptmodellingworkingseminarswere conducted.Agraphicrepresentationofconceptsandtheirrelationshipswasdevelopedtobridgethegap betweendifferentdisciplines.Aconceptmodelofresearchasaphenomenonwascreated,whichincluded atotalof18definedconceptsandtheirrelationships.Inasecondphasefourmainareasweredistilled, describingresearchpracticeinamultidisciplinarycontext:Credible,Contributory,Communicable,and Conforming.Eachofthesewasfurtherspecifiedinaconcepthierarchytogetherwithadefined terminol-ogy.Acomprehensivequalitymodelincluding32concepts,basedonthefourmainareas,wasdeveloped fordescribingqualityissuesofresearchpractice,wherethemodelofresearchasaphenomenonwas usedtodefinethequalityconcepts.Thequalitymodelmaybeusedforfurtherdevelopmentofelements, weightsandoperationalizationsrelatedtothequalityofresearchpracticeindifferentacademicfields.

1. Introduction 1.1. Background

Afundamentalquestionthatcanbeposedwithinanyﬁeldof researchis‘Whatconstitutesgoodorhighqualityresearch(or sci-entiﬁc) practice?’.This questionis relevant for researchboth in auniversitycontextandinanorganizationalorinnovation con-textforresearchanddevelopmentactivities.However,beforesuch criteriacanbeformulated,weneedareasonablycommon under-standingofwhatresearchitselfreallyis.

∗ Correspondingauthor.Tel.:+46852480000.

E-mailaddresses:[email protected](P.Mårtensson),[email protected] (U.Fors),[email protected](S.-B.Wallin),[email protected] (U.Zander),[email protected](G.H.Nilsson).

1 _Tel.:₊₄₆₈₇₃₆₉₀_00. 2 _Tel.:₊₄₆₈₆₇₄_7479. 3 _Tel.:₊₄₆₇₀₂₁₉₉₁_90. 4 _Tel.:₊₄₆₈₇₃₆₉₀_00.

Scienceandresearchareontologicallychallenging,and previ-ous researchrevealsdifferentviewsand remains ambiguous.A recentdefinitionofsciencewasproposedbytheBritishScience Council:“Scienceisthepursuitofknowledgeandunderstandingof thenaturalandsocialworldfollowingasystematicmethodology basedonevidence”(ScienceCouncil,2009:www.sciencecouncil. org/definition). Based on somewhat similar definitions, several studieshaveexploredtheconceptofresearch.InthisrespectIsrael (2005) acknowledged and explored the complexity of science, Patton(1990)mentionedthatitisimportanttoidentifythepurpose ofresearch,andGalletal.(1996)discussedhowresearchmight contributeinthefieldofeducation.Inthemedicaldomain,Grinnell (1990)arguedthattheendingsofclinicalresearchprotocolsareof importanceindistinguishingtherapyfromresearch.In2000the sameauthorstatedthattheeverydaypracticeofscienceis nei-therrealismnorsocialconstructivism,butratherisbalancedon acontextualledgebetweenthetwo,andsaidthatheconsidered discoveryandcredibilitytobethetwocentralfeaturesofresearch (Grinnell,2000).Ulrich(2006)hasanalyseddifferenttrapsthatare currentlycommonandthatleadtoasomewhatlimitedreflective http://dx.doi.org/10.1016/j.respol.2015.11.009

(2)

researchpractice,and he describesa rethinkingapproach. Fur-ther,Quaye(2007)arguesforextendingwhatcountsasresearch withinthesocialsciencessothatitismorelikelytoinclude dif-ferentmethodologiesandwritinggenres.Nickelsen(2009),in a similarapproach,supportsthenotionofinterventionistresearch thatisnotjustfocusedonsimpleone-waycausationintheﬁeld thatisbeingstudied.Inparallelwiththis,therehasbeenongoing discussionaboutrethinkingknowledgeproductioningeneral(e.g. HesselsandvanLente,2008;Tsaoetal.,2008).Inthisnewmode ofknowledgeproduction,oftenreferredtoasMode2,knowledge isproducedinthecontextofanapplication(Gibbonsetal.,1994). Knowledgecanbeproducedindifferentcontexts,andtheconcepts of‘knowinginaction’(AminandRoberts,2008)and‘situated learn-ing’(LaveandWenger,1991)highlighttheimportanceofavariety ofcontextualfactors.Itisimportanttokeepthisinmind,notleast inthelightoftheconsiderableamountofknowledgeproduction takingplaceinResearchandDevelopment(R&D)departmentsin companies.

In summary, there is broad criticism of theso-called linear modelofscience,anditisarguedthatconceptssuchasintuition andpassionhavebecomejustasimportantasobjectivityandlogic (Dash,2009;Grinnell,2009),andthereareseveralverydifferent viewson, anddeﬁnitions of,researchpractice.In ourpaperwe thereforeconcentrateoureffortsonworkingtowardsageneric def-inition(ormodel)ofwhatresearchis.Then,basedonthismodel, itmaybepossibletodeﬁnethegenericcomponentsofqualityof researchpractice.

However,beforedescribinganddiscussingourstudyandthe resultingmodel,weneedtoclarifytheterminologyused.Insome publicationsandwebsitesonthistopic,thereseemstobesome confusionbetweenthetermresearch andthetermscience, and thesetermsseemtobeusedinterchangeably.Inourviewtheterm scienceisbroader,andresearchismorelikethepracticeof work-inginascientiﬁcmanner.Researchiswhatyoupractise,andthe resultofthisworkisscience.Wehaveusedthetermsresearchand researchpracticethroughoutthispaper,asthescopeofourstudy comprisestryingtodeﬁnewhathighqualityscienceproduction mightbe.

Moreover,astheevaluationofresearchpracticeisoneofour end-goals,wemayalsoneedtodefinewhatwemeanby evalua-tion.Inourview,thepracticeofevaluationcanbedefinedasan activityinwhichcertainaspectsofthequalityofresearchpractice areinvestigated.Butwhatdoesthisreallymean?Theambitionto evaluateresearchhasalonghistorythatisfulloftensions, ambigu-itiesandmisunderstandings.Somecountrieshaveformednational commissionsforevaluatingresearch,whichseemtofocuson bib-liometricanalysestomeasureresearchquality(Jiménez-Contreras etal.,2003),buttheevaluationofresearchmayincludemanyother aspects.Thecurrentdebateisforexamplehighlightingtheproblem ofhavingevaluations“ledbythedataratherthanbyjudgement” (Hicks etal.,2015:429).Anoftenciteddefinitionof evaluation is“...aprocessforcollectingandsynthesizingevidencethatcan makeconclusionsaboutthestateofaffairs,value,merit,worth, significanceorqualityofaprogram,product...”(Mathison,2005), whichimpliesthatevaluationcanusenumerousmethodsand mea-sureawidevarietyofaspects(seealsoMertens,2015).

1.2. Dimensionsofthequalityofresearchpractice

Evaluationofthequalityofresearchpracticeisatruly impor-tantissueinmostscientiﬁcdomainsandatmanylevels(European ScienceFoundation,2012).Increasingly,wearealsoseeingthese assessment efforts across disciplinary and national boundaries. Moreorlesselaborateeffortshavebeenmadeinrecentyearsto evaluatethequalityofresearchpracticeinahostofdifferent sett-ings.Theseeffortsaffectresourceallocation,scientiﬁcactivity,and

theverylivesofresearchersacrosstheglobe.Qualityisthefocusfor severaldifferentreasons,andisexaminedinavarietyofcontexts suchasintheevaluationof:

–researchgrantapplications

–researchmanuscriptsandpublications –speciﬁcresearchtopics

–researchgroupsandconstellations –institutions

–nationalsystemsforproducingscienceandinnovation

Regardingtheissueofmeasuringthequalityofresearchinthe widerscientificcommunity,itisdifficulttofindauniversal defini-tionofwhatconstitutesgoodscientificpractice.Thefocusatsome universitiesisonlyonthenumberandqualityofpublicationsin scientificjournals,whereasotherinstitutionsfocusonallkindsof publications.However,inanincreasingnumberofacademicfields it isbecoming moreand morecommonfor scientificoutputto bemeasuredinwaysotherthansimplycountingthenumberand qualityofpublications.

Severalcostlyquality-assessmentprojects have been under-taken lately toimprove the quality of research practiceat the authors’ own institutions, to determine which research areas shouldreceive funding, tofindout whetherand where quality improvementsarenecessary,andtobenchmarkthequalityofa certaininstitutionagainstthatofleadinginternationalinstitutions. However,theavailablescientificliteratureonresearchquality,and onwhatcanreallybedefinedasresearch,isscarce.Some exam-plesdoexist.InItaly,forexample,nationalreferenceguidelinesfor theevaluationofresearchpracticehave,ingeneral,advocatedan approachthatincludessocio-economicimpact,resourceattraction andresourcemanagementascriteria(CIVR,2006).IntheUS,the criteriaforevaluatingresearchgrantapplicationsattheNational InstitutesofHealthincludeshortdefinitionsoffiveconcepts: signif-icance,approach,innovation,investigators,andenvironment(NIH, 2008).Inarecentevaluationofresearchconstellationswithina largeuniversityinSweden,thequalityofresearchpracticewas measuredbyconsideringtheattentionreceivedconcerningthe sci-entific,technological,clinicalandsocio-economicsignificanceof theirpublications,includingtheimplementationofresearchresults insociety(ExternalResearchAssessment(ERA),2010).InSweden today,however,thereseemtobeatleastasmanywaystomeasure whatconstitutesa goodscientificstudyorpublication,asthere areresearchinstitutions.InCanada,standardqualityassessment criteriaforresearchpapershavebeendeveloped,andthesedeal separatelywithquantitativeandqualitativeresearchstudies(Kmet etal.,2004).

However,itisnotourgoaltodistinguishsometypesof scien-tificmethodsthatareinherently‘good’fromothersthatmaybe ‘bad’.Ourcontentionisthatalmostanyscientificmethodcanbe appropriate,givenasoundresearchdesign.Itistheresearch ques-tion(s)athandthatshouldleadtothedecisiononwhichresearch designandmethod(s)shouldbeused,andqualitymaybehighas longasthemethodsareusedwithrigourandquality.Inourview, theoriescanbeseenas‘maps’andresearchmethodsas‘nets’;both arehighlycontextdependentinhowtheyfindandcapturethe elementsforproducingnewknowledge.

Qualityassuranceandevaluationmeasuresaremeanttobeas objective and reliableas possible.They generallyhave theaim ofincreasingawarenessaboutthecurrentstatusandstandingof theresearchthatisunderway.However,thegeneralproblemis thatnearlyalltherecentevaluationprojectshaveuseddifferent measuresandweightsfortheappliedvariables,makingitdifﬁcult tocompareaninstitution’sevaluationresultswiththoseofother institutionsordisciplines.Speciﬁcexamplesofproxyvariablesthat

(3)

havebeenusedintheevaluationsofthequalityofresearchpractice mentionedaboveinclude:

–Publicationmeasures(e.g.number,qualityandimpact) –Numberandquality(academicdegrees)oftheresearchers

them-selves

–Sizeofnationalandinternationalscientiﬁcnetworks –Amountandnumberofexternalresearchgrantsreceived –Amountandnumberofintra-organizationalgrants –NumberofPhDthesesproduced

–Numberofpostdoctoralorguestresearchers

Thelackofwidelyacknowledgedqualitystandardsforresearch practiceissomewhatsurprising.Aconsequenceofitisthatjudges ofthequalityof research–universityboards,scholars,funding agencies,journaleditorsandjournal reviewers–applythe val-uesand standards of theirown minds,fields or disciplines.To applyone’sownvaluesispartoftheevaluationprocess,buthaving generallyacknowledgedqualitystandards,insteadofdeveloping idiosyncraticones,islikelytomakeiteasiertomakefair evalu-ations.Althoughmostqualityevaluationshavebeenperformed withinuniversities,privateandpublicresearch-oriented compa-niesandotherscientificinstitutionshavealsocarriedoutsimilar evaluations.Someofthequalityevaluationshithertohave encom-passedentireuniversities,whileothershave focusedoncertain disciplines like life sciences, information technology, and even morespecificresearchtopics likecancer, diabetesand life-long learning.

Afollowing fundamental questionistherefore howresearch carriedoutwithinaparticularfieldshouldbeevaluated.In addi-tiontothedifficultiesofjudgingcontent,itisacomplexprocessto attempttoapplyacertainevaluationmodelthathasbeenusedina specificsettingtoanothersubjectdomain,area,regionorcountry. Furthermore,itisnotnecessarilythecasethataspecificsetof eval-uationcriteriaandweightsthatworkwellin,forexample,medicine willworkaswellinthesocialsciencesorengineering.Although therehavebeenattemptstodevelopgeneralqualitycriteria(e.g. Lahtinenetal.,2005)andmethodsforreviewingevidencein spe-cifictopics(e.g.AlborzandMcNally,2004),therearestillgeneral challengesfacingmanyuniversities.Theremaybemanydifferent academicareas,facultiesanddepartments,allofwhichwant(or need)tobecomparedwithoneanother.

Inpreviousresearchonqualitydimensionsithasbeenshown thatmanymodelsdescribingthequalityofresearchpracticecanbe used(e.g.Gummesson,1991;Keen,1991;Mason,1996;Maxwell, 1996;Mårtensson,2003;MårtenssonandMårtensson,2007;Rubin andRubin,1995;Sutherlandetal.,1993),andthatdifferent sug-gestedsetsofdimensionsoftenoverlapindifferentways. Some criteria,suchasthoseofKleinandMyers(1999),focuson evaluat-ingaspecifickindofresearch.Theauthorspresentasetofprinciples “addressingthequalitystandardsofonlyonetypeofinterpretive research,namelytheinterpretivefieldstudy”(p.69).Ina simi-larfashion,DubéandParé(2003)discusspositivistcaseresearch. Othersarguethatcertaindimensionsaregenerallybettersuitedto somekindsofresearchthantoothers;forinstance,RubinandRubin (1995)arguethatvalidityandreliabilityarebettersuitedto quanti-tativeresearchastheyarenotappropriateforqualitativeresearch. Adistinctionisoftenmadebetweenrigourandrelevance(e.g.Keen, 1991),anditissometimesassumed,implicitlyorexplicitly,that thereis a trade-offbetweentheseconcepts.Robey andMarkus (1998)arguethatresearchersshouldstrivetoproduceresearch thatisbothrigorousandrelevant,andtheycallthisconsumable research.Inconclusion,specificcriteriaforevaluatingthequality ofresearchpracticeseemtobelacking,oratleasttheyarenot welldefined.Atthecoreoftheproblemisacontemporarydebate overwhethersocialscienceresearchmeetsthequalitycriteriaof

thenaturalsciencesintermsofcleardefinitionsofterminology, quantifiability,highlycontrolledconditions,reproducibility,and predictabilityandtestability(BerezowandHartsfield,2012).

Thereisthusaneedfordeterminingauniversalconceptmodel forthequalityofresearchpractice.Furtherworktoadaptandtest theuseandreliabilityofsuchamodelindifferentdomainsand disciplineswouldthenbedesirable.

1.3. Aims

Thegeneralaimofthisstudyistopresentanactionableand multidisciplinaryframework fortheevaluation ofthequalityof researchpractice,thatcanbeusedasaguideindifferentscientiﬁc ﬁelds.Giventhediversityandlackofuniformityincurrent assess-mentmethods,thisstudyseekstobringsomereliefbymakingan inventoryoftheelementsthatconstitutetheveryideaofresearch itself.

Theprimaryobjectiveoftheﬁrstphaseofthisstudywasto cre-ateamultidisciplinary,fundamentalconceptmodelofresearchas aphenomenon,or,inotherwords,todescribe‘whatresearchreally is’,andalsotodifferentiateresearchpracticefromotheracademic orprofessionalactivities.Thisconceptmodelwillbereferredtoas thereferencemodel.

Inthesecondphasetheobjectivewastocreatea multidisci-plinaryandfundamentalconcepthierarchyfortermsassociated withthequalityofresearchpractice,orinotherwords‘whatthe quality ofresearch practiceis andhow it canbedeﬁned’.These conceptswerethoughttobeusefulasastructureforidentifying importantdimensionsofresearchqualityacrossvariousﬁelds.This concepthierarchywillbereferredtoasthequalitymodel. 1.4. Limitationofscope

Thefocusinthisprojectisonthequalityofresearchpractice, andthusonrelevantaspectsofthequalityofresearchprocesses (thatis,aspectsofresearchprocessesthatonewould expectto bedescribedandthatarerelevantundertheapplicablerulesfor descriptions).Thismeansthattheaimisnottotrytocaptureall aspectsof theactualresearchprocess,which couldinclude,for example,frustration,perceivedtimepressure,numerousrewriting iterations,etc.Althoughtheseaspectsarelikelytooccuroftenina researchprocess,themainfocushereis,rather,onaspectsofthe qualityofresearchpractice,whichinturnhaveimplicationsonthe outcomeofthisprocess.Moreover,ouraimisnottocover devel-opmentpractice,orthe‘D’inR&D(seeSection4.3foradiscussion onthistopic).

2. Methodologicalapproaches 2.1. Workinggroup

A workinggroup thatincluded foursenior researchers with extensiveacademicexperienceandonesenior modellingexpert wasformedwithinanetworkofSwedishuniversities.Thegroup members had experience in various research ﬁelds, including medicine,dentistry,computerscience,systemssciences,social sci-ences,educationalresearch,healthcareinformatics,management, strategy,internationalbusinessandbusinessmodelling.Allofthe researchersinthegroupalsohadexperienceofdifferent universi-tiesinEurope,theAmericasandAustralasia,andhadmovedinto newresearchareasduringtheircareers.Intotaltheyhadmore than century’s worth of relevant experiencefrom 27 European universities,eightAmericanuniversities,andnineuniversitiesin Australasia.

Inordertostructurethediscussionsandtheworkofthegroup, concept modelling and an analysis of the concepts related to

(4)

researchwereperformedduringmodellingworkshopsledbythe modellingexpert. Initially, thegoalfor the workinggroupwas deﬁned, and broad limitsof themodel were setto include all relevantconceptsfordescribing and followingupresearchin a multidisciplinaryperspective.Theparticipantswereurgedtospeak freelyinordertocapturearichpictureofresearchandresearch pro-cesses(i.e.theperspectivewasthatoftheactiveresearcher).Eight half-dayworkshopswereheldduring2012–2015.

2.2. Conceptmodellingandrelatedstandards

Whenworkingtowardsthegoaloftryingtofindageneralmodel ofwhatresearchis,itwasconsideredimportanttodefine differ-entcriteria(orconcepts) thatcan describethephenomenon of ‘researchpractice’.Thesecriteriashouldmakeitpossibleto sepa-rateresearchfromotherpractices,suchas,forexample,journalism. Therefore, asan aidtodefining the criteria,concept modelling wasconsideredtobea possiblewayforward.Inmany respects conceptmodelsprovideaparticularlyinterestingfocuswhen look-ingatcomplexconceptslike‘research’.Theycanberegardedas systemsconsistingofseveralcomponents(concepts)andthe rela-tionsbetweenthese components(conceptrelations).Generally, thesemodelscangiveusabetterunderstandingofhowourwork andknowledgeareorganized,and theyoftenreveal opportuni-tiesforsimplificationandforidentifyingproblemswhendifferent professionsareinteracting.Theyarefundamentalto information-basedorganizationsand areofspecialimportancetospecialists (Nuopponen,1994).

Conceptmodelshavebeendevelopedinseveralprojectswhere therehasbeenaneedtomapconceptsandknowledge.InSweden, forinstance,theNationalBoardofHealthandWelfarehasdeﬁned theconceptsof healthcareand social servicesby usingconcept modelling,andtheseconceptsarenowusedbytheentire health-caresector.Therearealsoothertypesofmappingavailable;for example,cognitivemappinghasbeenused,interalia,tounderstand whathealthcarejournalsrepresent(Shewchuketal.,2006).Inthe contextofthisparticularstudy,andpartlybasedonthese expe-riences,wedecidedthatconceptmodellingwouldbea suitable approach.

Theapproachtoconcept modellingusedhereistofocuson themeaningofsalientconceptsbydefiningthemwith relation-ships.ThismethodoriginatesfromStockholmUniversity(Boman etal.,1997),buthasbeenfurtherdevelopedbyapplyingittomany differenttypes ofprojects—forexample,todescribe meta mod-elsin standardization(ISO/IEC/IEEE42010:2011,2011).Another commonlyusedmethodfordescribingtheknowledgeofadomain, whichwechosenottouse,isby‘conceptmaps’.Thesedifferfrom conceptmodelsintwomainways.Aconceptmapdoesnotuse cardinality,whichisessentialfordefiningaconceptinaformally stringentway.Conceptmodellingalsotriestopinpointbasic con-cepts(oftencalledatomicconcepts)thatconstitutethefoundation forexpressingdifferentstatementsinthedomain,ratherthanjust connectingthewordsused.Inthiswayconceptmodelsareboth morestringentandmoreflexiblethanconceptmaps,whichiswhy webelievedthatconceptmodellingcouldbeausefultooltouse fordeterminingwhatresearchreallyis.

Themethodappliedinthestudyusesgraphicrepresentations oftheconceptsandtheirrelationshipswithotherconcepts, denot-ingattributesthatconstitutetheirmeaning.Thismethodhasbeen usedsuccessfullyin differentbusinessdomains todescribe and deﬁnethebasicconceptsofabusiness(Hedinetal.,2007), and asa foundationforbusinessdevelopmentbydeﬁning terminol-ogyandITrequirements.Themethodologyisbasedonathorough understandingofterminology,philosophyandlogic.

Itisimportanttodistinguishthefundamentalmethod,whichis todeﬁneconceptsbasedontheirrelationshipswithotherconcepts,

Fig.1. Exampleofnotationofconceptsandcardinality.

fromthegraphicalrepresentationsusedtonotatetheconcepts. Thenotationusedinthisstudyusestheclassdiagramsubsetof UMLversion2.1(http://www.uml.org)insteadoftheConceptual ModellingLanguage(CMOL)(Bomanetal.,1997).Thereasonfor thisisthewideinternationalunderstandingandapplicationsof UML.TheclassdiagramnotationofUMLhasbeendevelopedto modelinformation,butcanbeusedwithoutsemanticlosstomodel conceptsasdescribedinanISOstandardforterminologymodelling (ISO/TR24156:2008).

2.3. Conceptmodellingandgraphicalrepresentations

Inthismethodanditsnotation,conceptsarerepresentedas rec-tanglesandassociations(attributes)bylinesandarrows,asshown inFig.1.Speciﬁcconceptsarewritteninitalicsinthetext,andthe nameoftheattributeistobereadinthedirectionofthearrow(the triangleabovetheassociationline),asforexample‘apersonowns acar’,asalsoseeninFig.1.

Byusing cardinalitywe canmakethemodel morestringent byspecifyinghowtheelementsofthetwoconceptsinterrelateor associate,intermsof:

–1oneandonlyone –0.1zeroorone –1.*oneormany –*zeroormany

Anotationusingtheexampleofapersonandacarisshown inFig.1,and shouldberead‘A personownsoneand onlyone car’.Inconceptmodellingthismeansthattherecannotbea Per-sonwhodoesnotownaCar,andthataPersoncannotownmore thanoneCar(deﬁnedtermsarecapitalisedthroughoutthearticle). Thisis,ofcourse,nottheusualcase,butitcanbetheruleinthe contextdescribedbythemodel.Onecould,ofcourse,askwhether thenameoftheconceptPersonisrelevant;perhapsitshouldbe labelledOwnerofCarinstead.Thisisthekindofquestionthatis discussedinatypicalworkshopusingconceptmodelling.Thisis alsothetypeofquestionthatwasdiscussedduringtheworkshops intheprojectreportedhere.

TheexampleinFig.2illustratesthatResearchemanatesfromat leastoneQuestionatHand,andaimsforatleastonepieceofNew Knowledge.Accordingtoourdefinition(conceptmodel),you can-notcallsomethingResearchifitisnotaimingforNewKnowledge anddoesnotemanatefromaQuestionatHand.Thisisthewaywe definetheconceptinconceptmodelling,andthissmallexample onlyshowstwoconceptsdefiningResearch,andisnotintendedto beexhaustive.Inthecompleteconceptmodel,weseeseveral con-tributingconceptsdefiningResearch.Wedescribedallconceptsby theirterms,definitions,anddifferentrelations(genericor associa-tive).Intheconceptmodelallconceptsarelabelled,andaredefined bytheirrelationships.

2.4. The‘twomodelapproach’

Theconceptmodelis,asmentionedearlier,usedasareference modelthatdescribeswhatresearchisinaneutralway,andcanalso beseenasacommonlanguageaboutresearch.Thisreferencemodel enablesustoreasonaboutqualityaspectsinamethodicalway, byreusingtheconceptsandtheirinherentdeﬁnitionsinquality

(5)

Fig.2. Exampleofnotationofresearch-relatedconceptswithcardinality.

Fig.3. Relationbetweentheconceptmodelandthequalitystatementsintheconcepthierarchy.

statements.Bydescribingthequalitystatementsinthiswaywecan ensurethatthequalitystatementsarecomparableandcoherent accordingtothereferencemodel.InternallyValid,forexample,is describedas:“AcorrectScientiﬁcMethod(incl.researchdesign)is usedinrelationtoQuestionatHandandContext,andNewKnowledge isProvable”,seeFig.3.

3. Results

3.1. Phase1:Conceptmodelofresearch

Inthecomprehensiveconceptmodelofresearch,thereference model,wefoundatotalof18concepts,andrelationshipsbetween themthatwererelatedtotheconceptsanddeemedrelevantto theaims of ourstudy (Fig. 4).The model containedtwo types ofrelationship: onegeneric(is-a)relationshipand anumber of ontological(e.g.describes)relationships,andtheconceptsinthe modelweredeﬁnedbytheirrelationships.Themodelfocuseson theconceptResearchandassociatedconceptsinordertodeﬁnethe

meaningofresearch.Inordertomakethiscomprehensiveconcept modeltransparentandwell-deﬁned,allconceptsusedinthemodel aredescribedinTable1.

Adeﬁnitionofresearchthatcanbededucedfromthereference modelis:‘ResearchisaConsciousActionthataimsforNew Knowl-edge,emanatesfromoneorseveralQuestionsatHand,studiesone orseveralContexts,buildsuponExistingKnowledge,usesoneor sev-eralScientiﬁcMethods,isdocumentedinoneDescribedProcedure, requiresTransparencyandrelatestooneorseveralSystemsofRules’.

3.2. Phase2:Concepthierarchyofresearchqualityandrelated deﬁnedterms

Asasecondphase,theconceptmodelofresearchwasusedas abasisfordetermininganddiscussingdifferentaspectsofquality andevaluationinresearch.Withthisprocedure,important qual-ityaspectsthatcouldberelatedtotheconceptofresearchwere deﬁned.

(6)

Fig.4. Conceptmodelofresearch.

Fourmainaspectsemergedfromtheworkshopsheldaspartof thisproject,andtheyarelargelyconsistentwithearlierresearch. Previousresearchhaspointedout theﬁrstthree aspects, credi-ble,contributoryandcommunicable(Mårtensson,2003;Mårtensson andMårtensson,2007).Duringtheworkshopsinthisproject, how-ever,itwasfoundthatthesethreeaspectsdidnotcaptureallthose thatemergedinthediscussions.Throughthemodellingapproach a fourth aspectemerged, conforming. By conforming, we mean researchthatisalignedwithregulations,ethicalandsustainable.

The concept hierarchy deﬁned during the workshops thus breaksdownintofourmainareaslabelled‘Credible’,‘Contributory’, ‘Communicable’and‘Conforming’(Fig.5).These branches consti-tute themain dimensions of research quality according toour concept hierarchy, or quality model. Each of these wasfurther speciﬁeduntil32conceptswereestablished,including9branch

conceptsand23leafconcepts.Formaldefinitionswerethengiven toallconceptsinthequalitymodel,andallconceptsinthe ref-erencemodelfromthefirstphasewereneededfordefiningthese concepts.

FortheCrediblepartoftheconcepthierarchy,sixrelated con-ceptsweredeveloped(Table2).Itwaspossibletoformulateformal definitionsbasedonconceptsinthereferencemodelfromthefirst phase.Forexample,Reliablewasdefinedas:‘ThechosenScientific MethodisappropriateforthepresentQuestionatHandandContext, andisdocumentedinaDescribedProcedurethatotherscoulduse toreachasimilarresultinthesameContext’.

FortheContributorypartoftheconcepthierarchy,sevenrelated conceptsweredeveloped(Table3),andformaldeﬁnitionswere setupaccordingly.Forexample,CurrentIdeawasdeﬁnedas:‘The QuestionatHandisinaccordancewiththeContext’.

Table1

Deﬁnitionsofallconceptsintheconceptmodelofresearch.

Term Deﬁnition

Actor APersoninitiatingand/orperformingaConsciousAction Beneﬁcial ApositiveeffectofNewKnowledgeforaTargetGroup Consciousaction Aprocessinitiatedand/orperformedbyanActor.

Context AnenvironmentalorintellectualsettingwheretheResearchtakesplaceand/orisstudied,andwhereExistingKnowledgeisvalid Describedprocedure AdescriptionofhowtheResearchwillbeperformedanddocumentedaccordingtotheRulesforDescription

Existingknowledge KnowledgethatisbuiltonbytheResearch,existsinaContext,canbedocumentedinaSourceandisexpandedwithNewKnowledge Newknowledge KnowledgethatexpandsExistingKnowledge,isProvable,andisBeneﬁcialforaTargetGroup

Person Ahumanbeing

Provable EvidencethattheNewKnowledgeisdemonstrable Questionathand AresearchquestionthatisthebaseforResearch

Relationship ArelationbetweentwoConsciousActionsshowinghowthoseactionsinteract

Research AConsciousActionthataimsforNewKnowledge,emanatesfromoneorseveralQuestionsatHand,studiesoneorseveralContexts, buildsuponExistingKnowledge,usesoneorseveralScientiﬁcMethods,isdocumentedinoneDescribedProcedure,requires TransparencyandrelatestooneorseveralSystemsofRules

Rulesfordescription RulesdescribingwhataDescribedProcedureshouldcontain,includingitsintentionsandresults.ThiscandifferinregardstoContext, ScientiﬁcMethod,SystemofRules,ExistingKnowledgeandQuestionatHand

Scientiﬁcmethod AdescribedandprecisetechniqueusedforconductingtheResearch Source Documents,databasesorothermediathatcontainExistingKnowledge

Systemofrules Legalrequirements,regulations,normsandotherguidelinesthatinﬂuencehowResearchshouldbeperformed Targetgroup Individuals,organizations,enterprisesand/orsocietythatbeneﬁtfromNewKnowledge

(7)

Fig.5.Concepthierarchyofresearchquality.

Table2

DeﬁnitionsofconceptsrelatedtoCredible.

Term Deﬁnition

Credible ResearchthatisCoherent,Consistent,Rigorousand Transparent

Rigorous ResearchthatisContextual,InternallyValidand Reliable

Internallyvalid AcorrectScientiﬁcMethod(incl.researchdesign)is usedinrelationtotheQuestionatHandandContext, andNewKnowledgeisProvable

Reliable ThechosenScientiﬁcMethodisappropriateforthe presentQuestionatHandandContext,andis documentedinaDescribedProcedurethatothers couldusetoreachasimilarresultinthesameContext Contextual ExistingKnowledgethatisrelevantfortheContextis

used,andispresentedaccordingtoRulesfor Description

Consistent NewKnowledgeislogicallylinkedtoExisting KnowledgeandisinaccordancewiththeScientiﬁc MethodandQuestionatHand

Coherent AdequateconsiderationisgiventoExistingKnowledge inthechosenContext

Transparent RelevantNewKnowledgeinthereportingofresearch resultsisincludedandtheprocessisdescribedin relationtotheQuestionatHand,ScientiﬁcMethodand ExistingKnowledge

FortheCommunicablepartoftheconcepthierarchy,five con-cepts, with formal definitions, were developed (Table 4). For example,Readablewasdefinedas:‘CorrectlanguageintheResearch DocumentationfortheTargetGroup’.

FortheConformingpartoftheconcepthierarchy,fiverelated concepts,withformaldefinitions,weredeveloped(Table5).For example,Morally Justifiablewasdefined as: ‘TheResearch com-plieswithcurrentlyapplicableethicalstandardsasdescribedin the System of Rules’. All concepts were, as in the other parts, textuallydefined byusing conceptsfromtheconceptmodel of research.

Table3

DeﬁnitionsofconceptsrelatedtoContributory.

Term Deﬁnition

Contributory ResearchthatisOriginal,RelevantandGeneralizable Original ResearchthathasanOriginalIdea,usesanOriginal

ProcedureandproducesanOriginalResult Originalidea TheQuestionatHandhasnotbeenaskedbeforein

thecurrentContextorisinterpretedinanovelway Originalprocedure DescribedProcedureisoriginalinrelationtothe

QuestionatHand

Originalresult NewKnowledgeisProvableinrelationtoExisting Knowledge

Relevant ResearchthathasaRelevantResearchIdea, ApplicableResultandCurrentIdea

Relevantresearchidea QuestionatHandisrelevantforthecurrentTarget Group

Applicableresult NewknowledgeisBeneﬁcialforthecurrentTarget Group

Currentidea TheQuestionatHandisinaccordancewiththe currentContext

Generalizable NewKnowledgeispracticallyortheoreticallyuseful inContextsotherthantheonestudied

Table4

DeﬁnitionsofconceptsrelatedtoCommunicable.

Term Deﬁnition

Communicable ResearchthatisConsumable,Accessibleand Searchable

Consumable ResearchthatisStructured,Understandableand Readable

Structured TheResearchdocumentationfollowstheRulesfor Description

Understandable ThelanguageintheResearchdocumentationis understandablefortheTargetGroup

Readable CorrectlanguageintheResearchdocumentationfor theTargetGroup

Accessible NewKnowledgeiseasilyavailabletotheTargetGroup Searchable ThedocumentedNewKnowledgeisstructured

accordingtotheRulesforDescriptionandeasilyfound bytheTargetGroup

(8)

Table5

DeﬁnitionsofconceptsrelatedtoConforming.

Term Deﬁnition

Conforming ResearchthatisRegulatoryAligned,Ethical andSustainable

Alignedwithregulations TheResearchcomplieswithcurrently applicablelegalaspectsoftheSystemofRules Ethical TheResearchisMorallyJustiﬁable,Openand

supportsEqualOpportunities Morallyjustiﬁable TheResearchcomplieswithcurrently

applicableethicalstandardsasdescribedinthe SystemofRules

Open TheResearchdemonstratesTransparencywith currentlyapplicableethicalstandardsas describedintheSystemofRules Equalopportunities TheResearchisconsistentwithequal

treatmentaccordingtotheSystemofRules Sustainable TheResearchcomplieswithsustainable

developmentaspectsasdescribedinthe SystemofRules

4. Discussion 4.1. Mainﬁndings

Thisstudy presents a general,generic and multidisciplinary frameworkfor identifyingwhat thequalityofresearchpractice mightbe,andhowthatmodelmightbeusedtoevaluateresearch quality.First,wecreatedaconceptmodel ofresearchasa phe-nomenon,ourreferencemodel.Thismodelincludedatotalof18 definedconceptsandtheirrelationships.Second,wecreateda con-cepthierarchy,ourqualitymodel,with32conceptswithadefined terminology.Based onthisconcepthierarchy,thedifferent ele-mentscanbeusedeither withequalweights allocatedtoeach concept,orbyallocatingdifferentweightsaccordingtopreferences inthespecificcontextforevaluatingresearchquality.Thismeans thatintheapplicationofourcomprehensivequalitymodelsome aspectsmayevenbeallocatedtheweightofzero,creatingaspecial caseofourmodelwherecertainaspectsarenotbeincludedinthat specificcontextand/orsituation.Thepurposeofourqualitymodel istoofferabroadcomprehensivemodel,encompassingothermore specificcontextualsituations.

Ourintentioninthisstudyhasbeentoformulateaframework fortheassessmentofthequalityofresearchpractice.Weargue thatthisisausefulapproachfordiscussingresearchpracticeand itsqualityfrommanyperspectives,andcanhelptoadvance dis-cussions onresearchquality within and acrossdisciplines.Our intention,furthermore,isthatthisgeneralframeworkcanbeused asaplatformfordevelopingsubject-speciﬁcversionsofthe frame-work,whichcanbeﬁttedtoacontext.

4.2. Meaningoftheresultsandcomparisonwiththeliterature Ourfindingsregardingresearchasaphenomenoncannotbe eas-ilycomparedwithpreviousresearchas,toourknowledge,thisarea hassofarreceivedlimitedmodellingattention.Theconceptsinthe conceptmodelofresearchweredefinedusingintrinsicdefinitions, i.e.definitions basedonrelations tootherconcepts.These self-consistentdefinitionsareconsideredessentialinordertoachieve highreliabilitywhenusingthemodel.

Ourﬁndingsinthesecondphase,relatedtotheCredibleconcept, canreadilybecomparedwithpreviousworkbyShipman(1982) andGummesson(1991)thatsuggestsandintroducestheconceptof Transparency.However,itisimportanttorealizethatTransparency maybemoreorlesspronouncedincertainresearchpractices.For example,researchperformedata universityis oftenveryopen andtransparent,whileresearchperformedwithinacommercial companymaynotbe.Botharestillexamplesofresearchandthey

maybothbeofhighquality,butsomewheretheremightbealine betweenresearchleadingto‘proprietaryassets’,andacompany’s researchactivitiesthatresultinpublicationsandthusappearmore openandtransparent.SeealsothediscussioninSection4.5.

Inaddition,RubinandRubin(1995)emphasizeConsistencyand Coherenceinbothquantitativeandqualitativeresearch,whichis inlinewithourmodel.Lahtinenetal.(2005)alsoinclude Coher-enceasanimportantpartofscientiﬁcquality.Ourmodelissimilar totheworkofMårtenssonandMårtensson(2007)regardingthe introductionoftheconceptCredible;however,theconceptCoherent is added in ourmodel. Internal Validityand Reliability are con-ceptsthathavebeendiscussedfurtherby,forexample,Silverman (1993),aswellasbyRubinandRubin(1995).AlborzandMcNally (2004)alsostresstheimportanceofRigourwhenreviewing evi-denceregardingaccesstohealthcare.Keen(1991)and Maxwell (1996)haveemphasizedtheimportanceoffurtherperspectives forContextual,suchasspecifyingandanalysingaconceptual con-text,which is also in linewith thework of Aminand Roberts (2008).

TheContributorydimensioncanbecomparedtopreviouswork byMårtenssonandMårtensson(2007),wheretheconceptof Con-tributoryisused.OurconceptofRelevantResearchIdeais,however, somewhatdifferent.LikeBerthonetal.(2002),wefurther devel-opedOriginalbyusingthreeunderlyingconcepts.Incontrastto themodelbyBenbasatandZmud(1999),Accessibleinourmodel isnotapartofRelevance.Inasomewhatdifferentway,Rosemann andVessey(2008)divideRelevanceintothreedifferentdimensions (Importance,AccessibilityandSuitability).Further,AugierandMarch (2007)haveexploredRelevanceandseveralaspectsofusefulness inamoredetailedway.AlborzandMcNally(2004)alsoemphasize theimportanceofRelevancewhenreviewingresearchonaccessto healthcare.Inaddition,Vermeulen(2005)arguesthat synthesiz-ingRelevanceandRigourrequiresasystemicchangewhentryingto tacklearealpracticalquestionandsearchingforanswersina rigor-ousway.InlinewithMason(1996)andLeeandBaskerville(2003), wefoundGeneralizabilitytobebothpracticallyandtheoretically relevant.

In the Communicableconcept, we furtherexplored concepts usedbyMårtenssonandMårtensson(2007)byaddingthree sub-groupstoConsumable,which hasbeenarguedtobeofprimary importancebyRobeyandMarkus(1998)andDesouzaetal.(2006). ThesethreesubgroupsareStructured,UnderstandableandReadable, andtheyallconcernaspectsoftheresearchDocumentation.Inthe Communicableconcept,theothertwoconceptsareAccessibleand Searchable.Boththeseconceptsconcernhowtheresearch docu-mentationcanbeusedbythepotentialaudience,orTargetGroup. Itcanbearguedthatthispartofthemodelismorelikeameans toanendthananaspectofresearchitself.Atthesametime,itis verylikelythatitwillbedifﬁcultforresearchthatisnot consum-able,accessibleandsearchable,ornotcommunicable,toreachthe targetgroup,andthusthevalueoftheresearchcanbequestioned. OurﬁndingsonConformingaresomewhatnewcomparedto pre-viousconceptualexplorativeresearch.However,Pickersgill(2012) recentlyarguedthatsciencetodayisanethicalbusiness,that sci-enceandethicsareinmanywaysco-produced,andthatscienceand technologyscholarshavealonghistoryofdevelopingand imple-mentingbioethicsinpractice.InabroadsensethisimpliesaMoral andOpenorientationthatalsotakesintoaccountEqual Opportu-nitiesandiswellalignedwiththephilosophicalunderpinningsof academiaasa(rational)meritocracy.TheconceptsAlignedwith Reg-ulationsandSustainablearerarelymentionedinpreviousresearch, eventhoughbothshouldbeconsideredasessentialinallresearch practice.Theimportanceoftheconformingaspectsofresearchis highlightedbytheintensiveon-goingdebateonplagiarism, uneth-icalbehaviourandnorms,notonlyamongstudents,butalsoamong academics(e.g.HonigandBedi,2012).

(9)

4.3. Researchversusdevelopment

In industrial settings for research, the terms ‘research’ and ‘development’ (R &D) are oftenused together. This raisesthe questionofwhetherthereis adifferencebetweenthepractices ofresearchanddevelopment.Onecouldarguethatresearchand developmentare closelyrelated, which can bethecase, asthe developmentofaproductorservice isoftenbasedonprevious research.However,developmentdoesnot needtobebased on research,norisresearchalwayslinkedtothedevelopmentof some-thing.Thisisalsodiscussedintheliterature,where,forexample, Barge-GilandLópez(2014)clearlyseparateresearchfrom devel-opment.DiasandBresciani(2006)alsoseparateresearchactivities from developmentactivities within a company, indicating that thesemightberelated,butdonotneedtobe.Thebulkof devel-opmentproceduresperformedtodayareprobablyratherfarfrom research,inthesensethattheaimofdevelopmentisusually a (sellable)productorservice,buttheaimofresearchis(only)new knowledge.Therefore,researchperformedwithinacompanymay beseenassuchwhenithassomedegreeofopennessand trans-parency (such as when it is published) and is not only aimed atdevelopingaproductorservice.Therearealsolinksbetween companiesanduniversitiesconcerningdevelopmentand innova-tionactivities.Previousresearchhasshownthatdifferentsearch strategiesadoptedbycompaniesinﬂuencetheuseofuniversity knowledgeindevelopment(LaursenandSalter,2004).The inten-tionofourframework,however,shouldbeseenastodeﬁne(and lateron,toevaluate)researchpracticeonly,andnotdevelopment practice.

4.4. Anexampleasanillustrationofthemodel

If,then,weweregoingtoapplyourmodelofthecriteriaforhigh qualityresearchpractice,whatwouldthislooklike?Letusimagine alargeuniversitythatwantstoevaluatethequalityofitsresearch. Usingourapproach,averysimpliﬁedversionoftheprocesswould looksomethinglikethis:

(1)Selectionof anevaluationcommittee(preferably multidisci-plinary,primarilyforlegitimacyreasons).

(2)Discussion within the committee regarding the four over-arching concepts, credible, contributory, communicable and conforming,theunderlyingsub-concepts,andwhatthesemean forthemembers.

(3)Weighting of concepts, identifyingwhat theparticular uni-versity believes to be the most important concepts or sub-concepts.Forexample,thecommitteemightagreethatthe researchattheuniversityshouldreachouttotheworldina bet-terwaythanbefore,butthatitdoesnotneedtobegeneralizable tootherdomains.Inthiscase,adecisionmaybemadetogive Communicableanditssub-conceptsaweightof1.5andallthe otherconceptsaweightof1.0,exceptthatGeneralizablewould receiveaweightof0.5.

(4)Operationalizationofconcepts;collectionandanalysisofdata onresearchgroups’practicesorprojects;andanalysisofthe resultsbasedontheweightsofthedifferentconcepts.Thiswill probablycreatea fertilebasefora verylivelydiscussionon themeasures,weightsandendresults.Thetypeofdatathat shouldbecollectedwouldbecloselyrelatedtotheconcepts thathavebeenassignedthehighestweights.IfCommunicable hasbeenprioritized,thenprobablypublicationdataofvarious typesshouldbecollectedandscrutinizedintermsof,for exam-ple,numberofpublications,impactfactors,citationsetc.Butif Crediblehasbeengiventhehighestweight,itcouldbecome importanttoanalysedataonappliedresearchmethods.

4.5. Strengthsandlimitations

Itisimportanttounderstandthattheaimofthisstudyisto describewhatconstitutesresearch,andthentousethat knowl-edge to describe how research quality might be measured. To dothis,anumberofdifferentapproachescouldhavebeenused, includingbothqualitativeandquantitativemethods.Weuseda conceptmodellingapproach,assumingthatthesocialrealmmay besubjectedtothesamescientiﬁcmethodsofinvestigationand empiricismasthenaturalworld.Ourviewcouldperhapsbeseen asDurkheim(1898/1982),aswetrytoconstructconceptsinthe abstractinordertoformworkablecategoriesforempiricalanalysis. Wecouldhaveconsideredseveralqualitativeapproachesother thanconceptmodelling,suchasadatasamplebasedonastructured questionnaire,anopenquestionnaire,orafocusgroupinterview, followedbycontentanalysisandthendeterminationofthe var-iouscriteriaorconceptsthatcouldbederived fromthecontent analysis.However,webelievethat,withourliteraturereview,the opendiscussionsduringtheprocess,andthebroadrepresentation oftheresearchersintheteam,wehaveappliedamethodologically rigorousapproach.

Amorequantitativeapproachcould,forexample,haveincluded interviews,questionnairesfollowedbyconsensusmethodssuchas theDelphimethod,ethnographicﬁeldwork,ordiscourseanalysis. However,webelievethataquantitativeapproachwouldhavebeen lessappropriateatthisstage,becausetheconceptshadnotactually beendeﬁned,butthatsuchanapproachcouldverywellbeusedin afollow-upstudybasedonourmodel.

The study was carried out by a multidisciplinary group of authors,whohavebeeninvolvedinqualityassessmentsandthe evaluation of researchwithin medicine,life sciences, computer andsystemsciences,socialsciences,businessadministration,and economics.Theyhave experiencefromquality projectsat their respectiveuniversitiesandatotheruniversities(seeSection2.1for moredetail),allofwhichusedverydifferentdimensionsand vari-ablesintheirapproachestoresearchevaluation.Insomerespects ourworkcannotbegeneralized,asitisinitselfbasedonresults fromsocialconstructivistandqualitativeactivities.Incomparison torelatedstudies,however,itisbasedonamorerigorousmethod andapproach.

Weusedamultidisciplinaryapproachwhenmodelling,inline withouraims,whilebeingwellawarethatthisisa fundamen-tallydifferentwayofworkingfromtheapproachwhenconducting single-disciplinework.Unexpectedly,thiswasanobstacleto mod-elling at a more detailed level, as representatives of different disciplinessometimeshelddifferentviews.However,thebroad representationofresearchersfromseveraldifferentacademicareas isanimportantstrengthinrelationtothecredibilityofthemodel anditsfutureapplicability.Thebroadrepresentationofthe dif-ferentdomainexpertswhoparticipatedinthisstudyisthusan advantage,andtothebestofourknowledgeitdoesnotfeaturein relatedwork.However,weareawarethatevenifourexperience fromvariousacademicﬁeldsis broad,wecannotclaimthatwe coverallpossibleaspectsordisciplines.Specialistsinphilosophy orastronomy,forexample,mighthavereacheddifferentresults, although weargue thatourmodel would alsobevalidin such domains.Infutureresearch,however,thiscurrentstudycouldbe builtuponbyapplyingandtestingthegeneralandcomprehensive modelpresentedhereindifferentdisciplinarycontexts.

Concept modelling is a time-consuming process, and when experts from different domains are included, an initial learn-ing phase is necessary to reach a common ‘language’ and an understanding of themethod. Of course,concept modellingas suchcouldbediscussed,notonly inrelationtowhetherornot it is a good method to use in a context like this, but also in relation towhether thedeﬁned conceptsare well-deﬁned and

(10)

understandable.Asdescribedearlier,researchcanbeevaluatedin amultitudeofdimensions,oftenwithacertainoverlap.Further, thereisfrequentlyaneedforaformalconceptmodelandan estab-lishedterminologyinordertosupporttheevaluationofacertain topic.Variousapproachesinrelatedworkarecurrentlyconverging towardstheuseofspeciﬁedterms,basedonanunderlying for-malconceptmodel.Forexample,theSNOMEDCTmodeliswidely usedwithinhealthcare(SNOMEDCT,2014).Thereisconsequentlya needtoinvestigatetheconceptualcontentofthemultidisciplinary notionofresearchandrelatedqualityandevaluationissues. How-ever,developingatrulyconcept-orientedterminologyandmodel isademandingendeavour.

4.6. Implicationsforpolicymakers,researchersandfuture research

Ourmodelofresearchcanbeusefulinsituationsinwhichthere isaneedtodifferentiateresearchfromotherhumanactivitieswith socialaims—suchasreligion,artandjournalism.Forexample,when practicessuchashomeopathyhavebecomewidespread,and vari-ousdietsarefrequentlypromotedindifferentmedia,ourmodelcan beusedtocreateacleardistinctionbetweenclaimsbasedon gen-uinescientiﬁcstudiesandclaimsbasedonlyonbeliefsortraditions. Themodelcanbeseenandusedbothasatypeofchecklist(‘are someaspectsneglected?‘)andasapossiblefundamental struc-turefordiscussingthequalityofresearch.Theintentionwasfor thisgeneralcomprehensiveframeworktoformaplatformforthe furtherdevelopmentofcommonconcepts,termsandcriteriafor evaluationsofresearchqualitywithinandacrossspeciﬁcdomains, andtherebytocontributetoeffortstoimproveresearchqualityand understanding.

Furthermore,ourmodelof researchqualitycanbeuseful in severalacademicsituations.First, whenplanning,writing grant applicationsanddesigningresearch,theframeworkprovidesaset ofusefuldimensionstocoverandtoconsider.Second,the dimen-sionscanbeusefulinthedifferentadditionalstepsoftheresearch processinspeciﬁcprojects,forexampleindataanalysis.Third,it mayalsobeusedasaguidingprincipleindifferentevaluation pro-cessessuchasreviewprocesses.Fourthly,inmoregeneralquality workandtheassessmentorevaluationofconstellationsor insti-tutions,itmaybeofuseinaplethoraofsettings.Fifth,themodel canbeusedindoctoralprogrammesinwhich PhDstudentsare educatedandtrainedtoevaluatetheirownandothers’research activities.

Itisourhopeandcontentionthatthemodelpresentedhere maypromote understandingof thegeneric dimensionsof qual-ityinresearchpractice.Severalmorespecificapplicationsofour modelcouldcallforfurtherdevelopmentandadjustment,andalso forweightsfordifferentconceptsorpartsofourspecificsystem. Onecouldargue,forexample,thatitisverylikelythatresearchin medicinewoulduseweightsforsomeparametersthatwere dif-ferentfromtheweightsusedforresearchinlinguistics,computer scienceorsocialsciencestudies.Futurestudiesmaydevelopmore discipline-specificconceptmodelswithspecificweightsforeach concept.Furthermore,anapproachtotestinginter-evaluator reli-abilityinarealisticsituation,asintheCanadianstudymentioned above,would bedesirablein ordertocomeclosertoa science-orientedapproach to evaluating researchpractice (Kmet et al., 2004).Ofcoursethemodelproposedinthispaperdoesnotdelivera completedelineationofallthoughts,wordsandfactorsofrelevance toresearchpracticequality.Forexample,theconceptCurrentIdea couldbefurtherspecifiedandmodelledindifferentprofessions.

In sum, our model is expected to help us to represent and understandthecriteriaforhighqualityresearchinamore con-sistentmanner. It is tobeused both in itscurrent format and alsofor furtherdevelopmentof concepts,terms and criteria in

speciﬁcdomains.Wehopethatitwilltherebycontributetofuture effortstoimproveeveryday academicwork,linkingresearchers anddeveloping researchpractice,asassessmentand evaluation arefundamentalandincreasinglyimportantaspectsofallresearch practice.

5. Concludingremarks

Fromamultidisciplinaryperspective,acomprehensivequality modelincluding32concepts,basedonfourmainareas(Credible, Contributory,CommunicableandConforming)wasdevelopedfor describingthemaincontentofthequalityofresearchpractice.A modelofresearchasa phenomenon,witha totalof 18defined conceptsandtheirrelationships,wasneededtodefinethe qual-ityconcepts.Thequalitymodelmaybeusedasthestructurefor thefurtherdevelopmentofrelevantelements,weightsand opera-tionalizationsrelatedtothequalityofresearchpracticeindifferent academicfields.

Authors’contributions

GHNandPMconceivedanddesignedthestudy.SBWled,and alltheauthorsparticipatedin,themodellingwork.Alltheauthors wereinvolvedinwritingthemanuscript,andallofthemreadand approvedtheﬁnalmanuscript.

Acknowledgements

WewishtothanktheStockholmSchoolofEconomicsfor sup-portduringthemeetingsandworkshops.

References

Alborz,A.,McNally,R.,2004.Developingmethodsforsystematicreviewingin healthservicesdeliveryandorganisation:anexamplefromareviewofaccess tohealthcareforpeoplewithlearningdisabilities.Part2.Evaluationofthe literature—apracticalguide.HealthInf.Lib.J.21,227–236.

Amin,A.,Roberts,J.,2008.Knowinginaction:beyondcommunitiesofpractice. Res.Policy37,353–369.

Augier,M.E.,March,J.G.,2007.Thepursuitofrelevanceinmanagementeducation. Calif.Manage.Rev.49,129–146.

Barge-Gil,A.,López,A.,2014.RversusD:estimatingthedifferentiatedeffectof researchanddevelopmentoninnovationresults.Ind.Corp.Change,1–37. Benbasat,I.,Zmud,R.W.,1999.Empiricalresearchininformationsystems:the

practiceofrelevance.MISQ.23,3–16.

Berezow,A.B.,Hartsﬁeld,T.,2012.Whatseparatessciencefromnon-science?Real ClearSci.,2012/05/30.Availableatwww.realclearscience.com.

Berthon,P.,Pitt,L.,Ewing,M.,Carr,C.,2002.PotentialresearchspaceinMIS:a frameworkforenvisioningandevaluatingresearchreplication,extensionand generation.Inf.Syst.Res.13,416–427.

Boman,M.,BubenkoJr.,J.A.,Johannesson,P.,Wangler,B.,1997.Conceptual Modelling.UpperSaddleRiver,NJ,PrenticeHall.

CIVR(CommitteeforEvaluationofItalianResearch),2006.GuidelinesforResearch Evaluation,Availableat

http://vtr2006.cineca.it/documenti/lineeguidaEN.pdf.

Dash,D.P.,2009.Scienceasreﬂectivepractice:areviewofFrederickGrinnell’s book,everydaypracticeofscience.J.Res.Pract.5(ArticleR1).

Desouza,K.,ElSawy,O.,Galliers,R.,Loebbecke,C.,Watson,R.,2006.Beyondrigor andrelevancetowardsresponsibilityandreverberation:informationsystems researchthatreallymatters.Commun.AIS17,341–353.

Dias,R.T.,Bresciani,S.,2006.R&Dandknowledge:atheoreticalassessmentofthe internationalisationstrategies.Int.J.Technol.PolicyManage.6,1–32. Dubé,L.,Paré,G.,2003.Rigorininformationsystemspositivistcaseresearch:

currentpractices,trends,andrecommendations.MISQ.27,597–635. Durkheim,E.,1898/1982.TheRulesofSociologicalMethod.TheFreePress,New

York,NY.

EuropeanScienceFoundation,2012.Evaluationinresearchandresearchfunding organisations:Europeanpractices.In:AreportbytheESFMember OrganisationForumonEvaluationofPubliclyFundedResearch.,ISBN 978-2-918428-83-1.

ExternalResearchAssessment(ERA),2010.ExternalResearchAssessment. KarolinskaInstitutet,Stockholm,Availableatwww.ki.se.

Gall,M.D.,Borg,W.R.,Gall,J.P.,1996.EducationalResearch:AnIntroduction,sixth ed.Longman,WhitePlains,NY.

(11)

Gibbons,M.,Limoges,C.,Nowotny,H.,Schwartzman,S.,Scott,P.,Trow,M.,1994. TheNewProductionofKnowledge:TheDynamicsofScienceandResearchin ContemporarySocieties.SAGE,London.

Grinnell,F.,1990.Endingsofclinicalresearchprotocols:distinguishingtherapy fromresearch.EthicsHum.Res.12,1–3.

Grinnell,F.,2000.Thepracticeofscienceattheedgeofknowledge.ChronicleRev., March24.Availableathttp://chronicle.com.

Grinnell,F.,2009.EverydayPracticeofScience:WhereIntuitionandPassionMeet ObjectivityandLogic.OxfordUniversityPress,NewYork,NY.

Gummesson,E.,1991.QualitativeMethodsinManagementResearch.Sage Publications,NewburyPark,CA.

Hedin,A.,Jernberg,L.,Lennér,H.C.,Lundmark,T.,Wallin,S.-B.,2007.ToMeanand MeasureistheSameThing(Attmenaochmätasammasak).Studentlitteratur, Stockholm.

Hessels,L.K.,vanLente,H.,2008.Re-thinkingnewknowledgeproduction:a literaturereviewandaresearchagenda.Res.Policy37,740–760. Hicks,D.,Wounters,P.,Waltman,L.,deRijcke,S.,Rafols,I.,2015.TheLeiden

Manifestoforresearchmetrics.Nature520,429–431.

Honig,B.,Bedi,A.,2012.Thefoxinthehenhouse:acriticalexaminationof plagiarismamongmembersoftheAcademyofManagement.Acad.Manage. Learn.Educ.11,101–123.

ISO/TR24156:2008,2008.GuidelinesforUsingUMLNotationinTerminology Work,Availableathttp://www.iso.org.

ISO/IEC/IEEE42010:2011,2011.SystemsandSoftwareEngineering—Architecture Description,Availableathttp://www.iso.org.

Israel,G.,2005.Thescienceofcomplexity:epistemologicalproblemsand perspectives.Sci.Context18,479–509.

Jiménez-Contreras,E.,deMoyaAnegón,F.,DelgadoLópez-Cózar,E.,2003.The evaluationofresearchactivityinSpa.TheimpactoftheNationalCommission fortheEvaluationofResearchActivity(CNEAI).Res.Policy32,123– 142.

Keen,P.G.W.,1991.Relevanceandrigorininformationsystemsresearch: improvingquality,conﬁdence,cohesionandimpact.In:Klein,H.K.,Nissen, H-E.,Hirschheim,R.(Eds.),InformationSystemsResearch:Contemporary ApproachesandEmergentTraditions.IFIPElsevierScience,Philadelphia,PA, pp.27–49.

Klein,H.K.,Myers,M.D.,1999.Asetofprinciplesforconductingandevaluating interpretiveﬁeldstudiesininformationsystems.MISQ.23,67–94. Kmet,L.M.,Lee,R.C.,Cook,L.S.,2004.Standardqualityassessmentcriteriafor

evaluatingprimaryresearchpapersfromavarietyofﬁelds.In:HTAInitiative #13.AlbertaHeritageFoundationforMedicalResearch(AHFMR),Edmonton, AB,Availableathttp://www.ihe.ca.

Lahtinen,E.,Koskinen-Ollonqvist,P.,Rouvinen-Wilenius,P.,Tuominen,P., Mittelmark,M.B.,2005.Thedevelopmentofqualitycriteriaforresearch:a Finnishapproach.HealthPromot.Int.20,306–315.

Laursen,K.,Salter,A.,2004.Searchinghighandlow:whattypesofﬁrmsuse universitiesasasourceofinnovation?Res.Policy33,1201–1215. Lave,J.,Wenger,E.,1991.SituatedLearning:LegitimatePeripheralParticipation.

CambridgeUniversityPress,Cambridge.

Lee,A.S.,Baskerville,R.L.,2003.Generalizinggeneralizabilityininformation systemsresearch.Inf.Syst.Res.14,221–243.

Mårtensson,A.,Mårtensson,P.,2007.Extendingrigorandrelevance:towards credible,contributoryandcommunicableresearch.In:In15thEuropean ConferenceonInformationSystems(ECIS2007),StGallen,Switzerland. Mårtensson,A.,2003.ManagingMission-CriticalITintheFinancialIndustry.The

EconomicResearchInstitute,StockholmSchoolofEconomics,Stockholm. Mason,J.,1996.QualitativeResearching.SagePublications,London.

Mathison,S.(Ed.),2005.EncyclopediaofEvaluation.SagePublications,Thousand Oaks,CA.

Maxwell,J.A.,1996.QualitativeResearchDesign:AnInteractiveApproach.Sage Publications,ThousandOaks,CA.

Mertens,D.M.,2015.ResearchandEvaluationinEducationandPsychology: IntegratingDiversitywithQuantitative,Qualitative,andMixedMethods.Sage Publications,ThousandOaks,CA.

Nickelsen,N.C.,2009.Rethinkinginterventionistresearch:Navigatingoppositional networksinaDanishhospital.J.Res.Pract.5(ArticleM4).

NIH,2008.NationalInstitutesofHealthannouncesupdatedcriteriaforevaluating researchgrantapplications.In:NoticeNumber:NOT-OD-05-002,Availableat http://www.nih.gov.

Nuopponen,A.,1994.ConceptSystemsforTerminologicalAnalysis (Begreppsystemförterminologiskanalys).UniversityofVaasa,Vaasa. Patton,M.Q.,1990.QualitativeEvaluationandResearchMethods,seconded.Sage,

NewburyPark,CA.

Pickersgill,M.,2012.Theco-productionofscience,ethicsandemotion.Sci. Technol.Hum.Values37,579–603.

Quaye,S.J.,2007.Voiceoftheresearcher:extendingthelimitsofwhatcountsas research.J.Res.Pract.3(ArticleM3).

Robey,D.,Markus,M.L.,1998.Beyondrigorandrelevance:producingconsumable researchaboutinformationsystems.Inf.Resour.Manage.J.11,7–15. Rosemann,M.,Vessey,I.,2008.Towardimprovingtherelevanceofinformation

systemsresearchtopractice:theroleofapplicabilitychecks.MISQ.32,1–22. Rubin,H.J.,Rubin,I.S.,1995.QualitativeInterviewing:TheArtofHearingData.

SagePublications,ThousandOaks,CA.

ScienceCouncil,2009.Availableat:http://www.sciencecouncil.org.

Shewchuk,R.M.,O’Connor,S.J.,Williams,E.S.,Savage,G.T.,2006.Beyondrankings: Usingcognitivemappingtounderstandwhathealthcarejournalsrepresent. Soc.Sci.Med.62,1192–1204.

Shipman,M.,1982.TheLimitationsofSocialResearch.Longman,London. Silverman,D.,1993.InterpretingQualitativeData:MethodsforAnalysingTalk,

TextandInteraction.SagePublications,London.

SNOMEDCT,2014.Availableat:http://www.ihtsdo.org/snomed-ct/snomed-ct0/. Sutherland,H.J.,Meslin,E.M.,daCunha,R.,Till,J.E.,1993.Judgingclinicalresearch

questions:whatcriteriaareused?Soc.Sci.Med.37,1427–1430.

Tsao,J.Y.,Boyack,K.W.,Coltrin,M.E.,Turnley,J.G.,Gauster,W.B.,2008.Galileo’s stream:aframeworkforunderstandingknowledgeproduction.Res.Policy37, 330–352.

Ulrich,W.,2006.Rethinkingcriticallyreﬂectiveresearchpractice:beyondPopper’s criticalrationalism.J.Res.Pract.2(ArticleP1).

UniﬁedModelingLanguage(UniﬁedModelingLanguageTM_(UML®_)._Available_at

http://www.uml.org.

Vermeulen,F.,2005.Onrigorandrelevance:fosteringdialecticprogressionin managementresearch.Acad.Manage.J.48,978–982.