Evaluating research: A multidisciplinary approach to assessing research practice and quality

11  94  Download (0)

Full text

(1)

ContentslistsavailableatScienceDirect

Research

Policy

j ou rn a l h om ep a g e :w w w . e l s e v i e r . c o m / l o c a t e / r e s p o l

Evaluating

research:

A

multidisciplinary

approach

to

assessing

research

practice

and

quality

Pär

Mårtensson

a,1

,

Uno

Fors

b,2

,

Sven-Bertil

Wallin

c,3

,

Udo

Zander

d,4

,

Gunnar

H

Nilsson

e,∗

aDepartmentofManagementandOrganization,StockholmSchoolofEconomics,Sveavägen65,SE-11383Stockholm,Sweden bDepartmentofComputerandSystemsSciences,StockholmUniversity,POBox7003,SE-16407Kista,Sweden

cAlmainAB,LillaNorrby,SE-17891Munsö,Sweden

dDepartmentofMarketingandStrategy,StockholmSchoolofEconomics,Sveavägen65,SE-11383Stockholm,Sweden eDepartmentofNeurobiology,CareSciencesandSociety,KarolinskaInstitutet,SE-17177Stockholm,Sweden

a

r

t

i

c

l

e

i

n

f

o

Articlehistory: Received17June2014 Receivedinrevisedform 23November2015 Accepted24November2015 Availableonline24December2015 Keywords: Researchpractice Researchquality Researchevaluation Multidisciplinary Conceptmodel Rigour Relevance

a

b

s

t

r

a

c

t

Therearefewwidelyacknowledgedqualitystandardsforresearchpractice,andfewdefinitionsofwhat constitutesgoodresearch.Theoverallaimwasthereforetodescribewhatconstitutesresearch,andthen tousethisdescriptiontodevelopamodelofresearchpracticeandtodefineconceptsrelatedtoitsquality. Theprimaryobjectivewastoexploresuchamodelandtocreateamultidisciplinaryunderstandingofthe genericdimensionsofthequalityofresearchpractice.Eightconceptmodellingworkingseminarswere conducted.Agraphicrepresentationofconceptsandtheirrelationshipswasdevelopedtobridgethegap betweendifferentdisciplines.Aconceptmodelofresearchasaphenomenonwascreated,whichincluded atotalof18definedconceptsandtheirrelationships.Inasecondphasefourmainareasweredistilled, describingresearchpracticeinamultidisciplinarycontext:Credible,Contributory,Communicable,and Conforming.Eachofthesewasfurtherspecifiedinaconcepthierarchytogetherwithadefined terminol-ogy.Acomprehensivequalitymodelincluding32concepts,basedonthefourmainareas,wasdeveloped fordescribingqualityissuesofresearchpractice,wherethemodelofresearchasaphenomenonwas usedtodefinethequalityconcepts.Thequalitymodelmaybeusedforfurtherdevelopmentofelements, weightsandoperationalizationsrelatedtothequalityofresearchpracticeindifferentacademicfields.

©2015TheAuthors.PublishedbyElsevierB.V.ThisisanopenaccessarticleundertheCCBY-NC-ND license(http://creativecommons.org/licenses/by-nc-nd/4.0/).

1. Introduction 1.1. Background

Afundamentalquestionthatcanbeposedwithinanyfieldof researchis‘Whatconstitutesgoodorhighqualityresearch(or sci-entific) practice?’.This questionis relevant for researchboth in auniversitycontextandinanorganizationalorinnovation con-textforresearchanddevelopmentactivities.However,beforesuch criteriacanbeformulated,weneedareasonablycommon under-standingofwhatresearchitselfreallyis.

∗ Correspondingauthor.Tel.:+46852480000.

E-mailaddresses:par.martensson@hhs.se(P.Mårtensson),uno@dsv.su.se (U.Fors),sven-bertil.wallin@almain.se(S.-B.Wallin),udo.zander@hhs.se (U.Zander),gunnar.nilsson@ki.se(G.H.Nilsson).

1 Tel.:+4687369000. 2 Tel.:+4686747479. 3 Tel.:+46702199190. 4 Tel.:+4687369000.

Scienceandresearchareontologicallychallenging,and previ-ous researchrevealsdifferentviewsand remains ambiguous.A recentdefinitionofsciencewasproposedbytheBritishScience Council:“Scienceisthepursuitofknowledgeandunderstandingof thenaturalandsocialworldfollowingasystematicmethodology basedonevidence”(ScienceCouncil,2009:www.sciencecouncil. org/definition). Based on somewhat similar definitions, several studieshaveexploredtheconceptofresearch.InthisrespectIsrael (2005) acknowledged and explored the complexity of science, Patton(1990)mentionedthatitisimportanttoidentifythepurpose ofresearch,andGalletal.(1996)discussedhowresearchmight contributeinthefieldofeducation.Inthemedicaldomain,Grinnell (1990)arguedthattheendingsofclinicalresearchprotocolsareof importanceindistinguishingtherapyfromresearch.In2000the sameauthorstatedthattheeverydaypracticeofscienceis nei-therrealismnorsocialconstructivism,butratherisbalancedon acontextualledgebetweenthetwo,andsaidthatheconsidered discoveryandcredibilitytobethetwocentralfeaturesofresearch (Grinnell,2000).Ulrich(2006)hasanalyseddifferenttrapsthatare currentlycommonandthatleadtoasomewhatlimitedreflective http://dx.doi.org/10.1016/j.respol.2015.11.009

0048-7333/©2015TheAuthors.PublishedbyElsevierB.V.ThisisanopenaccessarticleundertheCCBY-NC-NDlicense(http://creativecommons.org/licenses/by-nc-nd/4. 0/).

(2)

researchpractice,and he describesa rethinkingapproach. Fur-ther,Quaye(2007)arguesforextendingwhatcountsasresearch withinthesocialsciencessothatitismorelikelytoinclude dif-ferentmethodologiesandwritinggenres.Nickelsen(2009),in a similarapproach,supportsthenotionofinterventionistresearch thatisnotjustfocusedonsimpleone-waycausationinthefield thatisbeingstudied.Inparallelwiththis,therehasbeenongoing discussionaboutrethinkingknowledgeproductioningeneral(e.g. HesselsandvanLente,2008;Tsaoetal.,2008).Inthisnewmode ofknowledgeproduction,oftenreferredtoasMode2,knowledge isproducedinthecontextofanapplication(Gibbonsetal.,1994). Knowledgecanbeproducedindifferentcontexts,andtheconcepts of‘knowinginaction’(AminandRoberts,2008)and‘situated learn-ing’(LaveandWenger,1991)highlighttheimportanceofavariety ofcontextualfactors.Itisimportanttokeepthisinmind,notleast inthelightoftheconsiderableamountofknowledgeproduction takingplaceinResearchandDevelopment(R&D)departmentsin companies.

In summary, there is broad criticism of theso-called linear modelofscience,anditisarguedthatconceptssuchasintuition andpassionhavebecomejustasimportantasobjectivityandlogic (Dash,2009;Grinnell,2009),andthereareseveralverydifferent viewson, anddefinitions of,researchpractice.In ourpaperwe thereforeconcentrateoureffortsonworkingtowardsageneric def-inition(ormodel)ofwhatresearchis.Then,basedonthismodel, itmaybepossibletodefinethegenericcomponentsofqualityof researchpractice.

However,beforedescribinganddiscussingourstudyandthe resultingmodel,weneedtoclarifytheterminologyused.Insome publicationsandwebsitesonthistopic,thereseemstobesome confusionbetweenthetermresearch andthetermscience, and thesetermsseemtobeusedinterchangeably.Inourviewtheterm scienceisbroader,andresearchismorelikethepracticeof work-inginascientificmanner.Researchiswhatyoupractise,andthe resultofthisworkisscience.Wehaveusedthetermsresearchand researchpracticethroughoutthispaper,asthescopeofourstudy comprisestryingtodefinewhathighqualityscienceproduction mightbe.

Moreover,astheevaluationofresearchpracticeisoneofour end-goals,wemayalsoneedtodefinewhatwemeanby evalua-tion.Inourview,thepracticeofevaluationcanbedefinedasan activityinwhichcertainaspectsofthequalityofresearchpractice areinvestigated.Butwhatdoesthisreallymean?Theambitionto evaluateresearchhasalonghistorythatisfulloftensions, ambigu-itiesandmisunderstandings.Somecountrieshaveformednational commissionsforevaluatingresearch,whichseemtofocuson bib-liometricanalysestomeasureresearchquality(Jiménez-Contreras etal.,2003),buttheevaluationofresearchmayincludemanyother aspects.Thecurrentdebateisforexamplehighlightingtheproblem ofhavingevaluations“ledbythedataratherthanbyjudgement” (Hicks etal.,2015:429).Anoftenciteddefinitionof evaluation is“...aprocessforcollectingandsynthesizingevidencethatcan makeconclusionsaboutthestateofaffairs,value,merit,worth, significanceorqualityofaprogram,product...”(Mathison,2005), whichimpliesthatevaluationcanusenumerousmethodsand mea-sureawidevarietyofaspects(seealsoMertens,2015).

1.2. Dimensionsofthequalityofresearchpractice

Evaluationofthequalityofresearchpracticeisatruly impor-tantissueinmostscientificdomainsandatmanylevels(European ScienceFoundation,2012).Increasingly,wearealsoseeingthese assessment efforts across disciplinary and national boundaries. Moreorlesselaborateeffortshavebeenmadeinrecentyearsto evaluatethequalityofresearchpracticeinahostofdifferent sett-ings.Theseeffortsaffectresourceallocation,scientificactivity,and

theverylivesofresearchersacrosstheglobe.Qualityisthefocusfor severaldifferentreasons,andisexaminedinavarietyofcontexts suchasintheevaluationof:

–researchgrantapplications

–researchmanuscriptsandpublications –specificresearchtopics

–researchgroupsandconstellations –institutions

–nationalsystemsforproducingscienceandinnovation

Regardingtheissueofmeasuringthequalityofresearchinthe widerscientificcommunity,itisdifficulttofindauniversal defini-tionofwhatconstitutesgoodscientificpractice.Thefocusatsome universitiesisonlyonthenumberandqualityofpublicationsin scientificjournals,whereasotherinstitutionsfocusonallkindsof publications.However,inanincreasingnumberofacademicfields it isbecoming moreand morecommonfor scientificoutputto bemeasuredinwaysotherthansimplycountingthenumberand qualityofpublications.

Severalcostlyquality-assessmentprojects have been under-taken lately toimprove the quality of research practiceat the authors’ own institutions, to determine which research areas shouldreceive funding, tofindout whetherand where quality improvementsarenecessary,andtobenchmarkthequalityofa certaininstitutionagainstthatofleadinginternationalinstitutions. However,theavailablescientificliteratureonresearchquality,and onwhatcanreallybedefinedasresearch,isscarce.Some exam-plesdoexist.InItaly,forexample,nationalreferenceguidelinesfor theevaluationofresearchpracticehave,ingeneral,advocatedan approachthatincludessocio-economicimpact,resourceattraction andresourcemanagementascriteria(CIVR,2006).IntheUS,the criteriaforevaluatingresearchgrantapplicationsattheNational InstitutesofHealthincludeshortdefinitionsoffiveconcepts: signif-icance,approach,innovation,investigators,andenvironment(NIH, 2008).Inarecentevaluationofresearchconstellationswithina largeuniversityinSweden,thequalityofresearchpracticewas measuredbyconsideringtheattentionreceivedconcerningthe sci-entific,technological,clinicalandsocio-economicsignificanceof theirpublications,includingtheimplementationofresearchresults insociety(ExternalResearchAssessment(ERA),2010).InSweden today,however,thereseemtobeatleastasmanywaystomeasure whatconstitutesa goodscientificstudyorpublication,asthere areresearchinstitutions.InCanada,standardqualityassessment criteriaforresearchpapershavebeendeveloped,andthesedeal separatelywithquantitativeandqualitativeresearchstudies(Kmet etal.,2004).

However,itisnotourgoaltodistinguishsometypesof scien-tificmethodsthatareinherently‘good’fromothersthatmaybe ‘bad’.Ourcontentionisthatalmostanyscientificmethodcanbe appropriate,givenasoundresearchdesign.Itistheresearch ques-tion(s)athandthatshouldleadtothedecisiononwhichresearch designandmethod(s)shouldbeused,andqualitymaybehighas longasthemethodsareusedwithrigourandquality.Inourview, theoriescanbeseenas‘maps’andresearchmethodsas‘nets’;both arehighlycontextdependentinhowtheyfindandcapturethe elementsforproducingnewknowledge.

Qualityassuranceandevaluationmeasuresaremeanttobeas objective and reliableas possible.They generallyhave theaim ofincreasingawarenessaboutthecurrentstatusandstandingof theresearchthatisunderway.However,thegeneralproblemis thatnearlyalltherecentevaluationprojectshaveuseddifferent measuresandweightsfortheappliedvariables,makingitdifficult tocompareaninstitution’sevaluationresultswiththoseofother institutionsordisciplines.Specificexamplesofproxyvariablesthat

(3)

havebeenusedintheevaluationsofthequalityofresearchpractice mentionedaboveinclude:

–Publicationmeasures(e.g.number,qualityandimpact) –Numberandquality(academicdegrees)oftheresearchers

them-selves

–Sizeofnationalandinternationalscientificnetworks –Amountandnumberofexternalresearchgrantsreceived –Amountandnumberofintra-organizationalgrants –NumberofPhDthesesproduced

–Numberofpostdoctoralorguestresearchers

Thelackofwidelyacknowledgedqualitystandardsforresearch practiceissomewhatsurprising.Aconsequenceofitisthatjudges ofthequalityof research–universityboards,scholars,funding agencies,journaleditorsandjournal reviewers–applythe val-uesand standards of theirown minds,fields or disciplines.To applyone’sownvaluesispartoftheevaluationprocess,buthaving generallyacknowledgedqualitystandards,insteadofdeveloping idiosyncraticones,islikelytomakeiteasiertomakefair evalu-ations.Althoughmostqualityevaluationshavebeenperformed withinuniversities,privateandpublicresearch-oriented compa-niesandotherscientificinstitutionshavealsocarriedoutsimilar evaluations.Someofthequalityevaluationshithertohave encom-passedentireuniversities,whileothershave focusedoncertain disciplines like life sciences, information technology, and even morespecificresearchtopics likecancer, diabetesand life-long learning.

Afollowing fundamental questionistherefore howresearch carriedoutwithinaparticularfieldshouldbeevaluated.In addi-tiontothedifficultiesofjudgingcontent,itisacomplexprocessto attempttoapplyacertainevaluationmodelthathasbeenusedina specificsettingtoanothersubjectdomain,area,regionorcountry. Furthermore,itisnotnecessarilythecasethataspecificsetof eval-uationcriteriaandweightsthatworkwellin,forexample,medicine willworkaswellinthesocialsciencesorengineering.Although therehavebeenattemptstodevelopgeneralqualitycriteria(e.g. Lahtinenetal.,2005)andmethodsforreviewingevidencein spe-cifictopics(e.g.AlborzandMcNally,2004),therearestillgeneral challengesfacingmanyuniversities.Theremaybemanydifferent academicareas,facultiesanddepartments,allofwhichwant(or need)tobecomparedwithoneanother.

Inpreviousresearchonqualitydimensionsithasbeenshown thatmanymodelsdescribingthequalityofresearchpracticecanbe used(e.g.Gummesson,1991;Keen,1991;Mason,1996;Maxwell, 1996;Mårtensson,2003;MårtenssonandMårtensson,2007;Rubin andRubin,1995;Sutherlandetal.,1993),andthatdifferent sug-gestedsetsofdimensionsoftenoverlapindifferentways. Some criteria,suchasthoseofKleinandMyers(1999),focuson evaluat-ingaspecifickindofresearch.Theauthorspresentasetofprinciples “addressingthequalitystandardsofonlyonetypeofinterpretive research,namelytheinterpretivefieldstudy”(p.69).Ina simi-larfashion,DubéandParé(2003)discusspositivistcaseresearch. Othersarguethatcertaindimensionsaregenerallybettersuitedto somekindsofresearchthantoothers;forinstance,RubinandRubin (1995)arguethatvalidityandreliabilityarebettersuitedto quanti-tativeresearchastheyarenotappropriateforqualitativeresearch. Adistinctionisoftenmadebetweenrigourandrelevance(e.g.Keen, 1991),anditissometimesassumed,implicitlyorexplicitly,that thereis a trade-offbetweentheseconcepts.Robey andMarkus (1998)arguethatresearchersshouldstrivetoproduceresearch thatisbothrigorousandrelevant,andtheycallthisconsumable research.Inconclusion,specificcriteriaforevaluatingthequality ofresearchpracticeseemtobelacking,oratleasttheyarenot welldefined.Atthecoreoftheproblemisacontemporarydebate overwhethersocialscienceresearchmeetsthequalitycriteriaof

thenaturalsciencesintermsofcleardefinitionsofterminology, quantifiability,highlycontrolledconditions,reproducibility,and predictabilityandtestability(BerezowandHartsfield,2012).

Thereisthusaneedfordeterminingauniversalconceptmodel forthequalityofresearchpractice.Furtherworktoadaptandtest theuseandreliabilityofsuchamodelindifferentdomainsand disciplineswouldthenbedesirable.

1.3. Aims

Thegeneralaimofthisstudyistopresentanactionableand multidisciplinaryframework fortheevaluation ofthequalityof researchpractice,thatcanbeusedasaguideindifferentscientific fields.Giventhediversityandlackofuniformityincurrent assess-mentmethods,thisstudyseekstobringsomereliefbymakingan inventoryoftheelementsthatconstitutetheveryideaofresearch itself.

Theprimaryobjectiveofthefirstphaseofthisstudywasto cre-ateamultidisciplinary,fundamentalconceptmodelofresearchas aphenomenon,or,inotherwords,todescribe‘whatresearchreally is’,andalsotodifferentiateresearchpracticefromotheracademic orprofessionalactivities.Thisconceptmodelwillbereferredtoas thereferencemodel.

Inthesecondphasetheobjectivewastocreatea multidisci-plinaryandfundamentalconcepthierarchyfortermsassociated withthequalityofresearchpractice,orinotherwords‘whatthe quality ofresearch practiceis andhow it canbedefined’.These conceptswerethoughttobeusefulasastructureforidentifying importantdimensionsofresearchqualityacrossvariousfields.This concepthierarchywillbereferredtoasthequalitymodel. 1.4. Limitationofscope

Thefocusinthisprojectisonthequalityofresearchpractice, andthusonrelevantaspectsofthequalityofresearchprocesses (thatis,aspectsofresearchprocessesthatonewould expectto bedescribedandthatarerelevantundertheapplicablerulesfor descriptions).Thismeansthattheaimisnottotrytocaptureall aspectsof theactualresearchprocess,which couldinclude,for example,frustration,perceivedtimepressure,numerousrewriting iterations,etc.Althoughtheseaspectsarelikelytooccuroftenina researchprocess,themainfocushereis,rather,onaspectsofthe qualityofresearchpractice,whichinturnhaveimplicationsonthe outcomeofthisprocess.Moreover,ouraimisnottocover devel-opmentpractice,orthe‘D’inR&D(seeSection4.3foradiscussion onthistopic).

2. Methodologicalapproaches 2.1. Workinggroup

A workinggroup thatincluded foursenior researchers with extensiveacademicexperienceandonesenior modellingexpert wasformedwithinanetworkofSwedishuniversities.Thegroup members had experience in various research fields, including medicine,dentistry,computerscience,systemssciences,social sci-ences,educationalresearch,healthcareinformatics,management, strategy,internationalbusinessandbusinessmodelling.Allofthe researchersinthegroupalsohadexperienceofdifferent universi-tiesinEurope,theAmericasandAustralasia,andhadmovedinto newresearchareasduringtheircareers.Intotaltheyhadmore than century’s worth of relevant experiencefrom 27 European universities,eightAmericanuniversities,andnineuniversitiesin Australasia.

Inordertostructurethediscussionsandtheworkofthegroup, concept modelling and an analysis of the concepts related to

(4)

researchwereperformedduringmodellingworkshopsledbythe modellingexpert. Initially, thegoalfor the workinggroupwas defined, and broad limitsof themodel were setto include all relevantconceptsfordescribing and followingupresearchin a multidisciplinaryperspective.Theparticipantswereurgedtospeak freelyinordertocapturearichpictureofresearchandresearch pro-cesses(i.e.theperspectivewasthatoftheactiveresearcher).Eight half-dayworkshopswereheldduring2012–2015.

2.2. Conceptmodellingandrelatedstandards

Whenworkingtowardsthegoaloftryingtofindageneralmodel ofwhatresearchis,itwasconsideredimportanttodefine differ-entcriteria(orconcepts) thatcan describethephenomenon of ‘researchpractice’.Thesecriteriashouldmakeitpossibleto sepa-rateresearchfromotherpractices,suchas,forexample,journalism. Therefore, asan aidtodefining the criteria,concept modelling wasconsideredtobea possiblewayforward.Inmany respects conceptmodelsprovideaparticularlyinterestingfocuswhen look-ingatcomplexconceptslike‘research’.Theycanberegardedas systemsconsistingofseveralcomponents(concepts)andthe rela-tionsbetweenthese components(conceptrelations).Generally, thesemodelscangiveusabetterunderstandingofhowourwork andknowledgeareorganized,and theyoftenreveal opportuni-tiesforsimplificationandforidentifyingproblemswhendifferent professionsareinteracting.Theyarefundamentalto information-basedorganizationsand areofspecialimportancetospecialists (Nuopponen,1994).

Conceptmodelshavebeendevelopedinseveralprojectswhere therehasbeenaneedtomapconceptsandknowledge.InSweden, forinstance,theNationalBoardofHealthandWelfarehasdefined theconceptsof healthcareand social servicesby usingconcept modelling,andtheseconceptsarenowusedbytheentire health-caresector.Therearealsoothertypesofmappingavailable;for example,cognitivemappinghasbeenused,interalia,tounderstand whathealthcarejournalsrepresent(Shewchuketal.,2006).Inthe contextofthisparticularstudy,andpartlybasedonthese expe-riences,wedecidedthatconceptmodellingwouldbea suitable approach.

Theapproachtoconcept modellingusedhereistofocuson themeaningofsalientconceptsbydefiningthemwith relation-ships.ThismethodoriginatesfromStockholmUniversity(Boman etal.,1997),buthasbeenfurtherdevelopedbyapplyingittomany differenttypes ofprojects—forexample,todescribe meta mod-elsin standardization(ISO/IEC/IEEE42010:2011,2011).Another commonlyusedmethodfordescribingtheknowledgeofadomain, whichwechosenottouse,isby‘conceptmaps’.Thesedifferfrom conceptmodelsintwomainways.Aconceptmapdoesnotuse cardinality,whichisessentialfordefiningaconceptinaformally stringentway.Conceptmodellingalsotriestopinpointbasic con-cepts(oftencalledatomicconcepts)thatconstitutethefoundation forexpressingdifferentstatementsinthedomain,ratherthanjust connectingthewordsused.Inthiswayconceptmodelsareboth morestringentandmoreflexiblethanconceptmaps,whichiswhy webelievedthatconceptmodellingcouldbeausefultooltouse fordeterminingwhatresearchreallyis.

Themethodappliedinthestudyusesgraphicrepresentations oftheconceptsandtheirrelationshipswithotherconcepts, denot-ingattributesthatconstitutetheirmeaning.Thismethodhasbeen usedsuccessfullyin differentbusinessdomains todescribe and definethebasicconceptsofabusiness(Hedinetal.,2007), and asa foundationforbusinessdevelopmentbydefining terminol-ogyandITrequirements.Themethodologyisbasedonathorough understandingofterminology,philosophyandlogic.

Itisimportanttodistinguishthefundamentalmethod,whichis todefineconceptsbasedontheirrelationshipswithotherconcepts,

Fig.1. Exampleofnotationofconceptsandcardinality.

fromthegraphicalrepresentationsusedtonotatetheconcepts. Thenotationusedinthisstudyusestheclassdiagramsubsetof UMLversion2.1(http://www.uml.org)insteadoftheConceptual ModellingLanguage(CMOL)(Bomanetal.,1997).Thereasonfor thisisthewideinternationalunderstandingandapplicationsof UML.TheclassdiagramnotationofUMLhasbeendevelopedto modelinformation,butcanbeusedwithoutsemanticlosstomodel conceptsasdescribedinanISOstandardforterminologymodelling (ISO/TR24156:2008).

2.3. Conceptmodellingandgraphicalrepresentations

Inthismethodanditsnotation,conceptsarerepresentedas rec-tanglesandassociations(attributes)bylinesandarrows,asshown inFig.1.Specificconceptsarewritteninitalicsinthetext,andthe nameoftheattributeistobereadinthedirectionofthearrow(the triangleabovetheassociationline),asforexample‘apersonowns acar’,asalsoseeninFig.1.

Byusing cardinalitywe canmakethemodel morestringent byspecifyinghowtheelementsofthetwoconceptsinterrelateor associate,intermsof:

–1oneandonlyone –0.1zeroorone –1.*oneormany –*zeroormany

Anotationusingtheexampleofapersonandacarisshown inFig.1,and shouldberead‘A personownsoneand onlyone car’.Inconceptmodellingthismeansthattherecannotbea Per-sonwhodoesnotownaCar,andthataPersoncannotownmore thanoneCar(definedtermsarecapitalisedthroughoutthearticle). Thisis,ofcourse,nottheusualcase,butitcanbetheruleinthe contextdescribedbythemodel.Onecould,ofcourse,askwhether thenameoftheconceptPersonisrelevant;perhapsitshouldbe labelledOwnerofCarinstead.Thisisthekindofquestionthatis discussedinatypicalworkshopusingconceptmodelling.Thisis alsothetypeofquestionthatwasdiscussedduringtheworkshops intheprojectreportedhere.

TheexampleinFig.2illustratesthatResearchemanatesfromat leastoneQuestionatHand,andaimsforatleastonepieceofNew Knowledge.Accordingtoourdefinition(conceptmodel),you can-notcallsomethingResearchifitisnotaimingforNewKnowledge anddoesnotemanatefromaQuestionatHand.Thisisthewaywe definetheconceptinconceptmodelling,andthissmallexample onlyshowstwoconceptsdefiningResearch,andisnotintendedto beexhaustive.Inthecompleteconceptmodel,weseeseveral con-tributingconceptsdefiningResearch.Wedescribedallconceptsby theirterms,definitions,anddifferentrelations(genericor associa-tive).Intheconceptmodelallconceptsarelabelled,andaredefined bytheirrelationships.

2.4. The‘twomodelapproach’

Theconceptmodelis,asmentionedearlier,usedasareference modelthatdescribeswhatresearchisinaneutralway,andcanalso beseenasacommonlanguageaboutresearch.Thisreferencemodel enablesustoreasonaboutqualityaspectsinamethodicalway, byreusingtheconceptsandtheirinherentdefinitionsinquality

(5)

Fig.2. Exampleofnotationofresearch-relatedconceptswithcardinality.

Fig.3. Relationbetweentheconceptmodelandthequalitystatementsintheconcepthierarchy.

statements.Bydescribingthequalitystatementsinthiswaywecan ensurethatthequalitystatementsarecomparableandcoherent accordingtothereferencemodel.InternallyValid,forexample,is describedas:“AcorrectScientificMethod(incl.researchdesign)is usedinrelationtoQuestionatHandandContext,andNewKnowledge isProvable”,seeFig.3.

3. Results

3.1. Phase1:Conceptmodelofresearch

Inthecomprehensiveconceptmodelofresearch,thereference model,wefoundatotalof18concepts,andrelationshipsbetween themthatwererelatedtotheconceptsanddeemedrelevantto theaims of ourstudy (Fig. 4).The model containedtwo types ofrelationship: onegeneric(is-a)relationshipand anumber of ontological(e.g.describes)relationships,andtheconceptsinthe modelweredefinedbytheirrelationships.Themodelfocuseson theconceptResearchandassociatedconceptsinordertodefinethe

meaningofresearch.Inordertomakethiscomprehensiveconcept modeltransparentandwell-defined,allconceptsusedinthemodel aredescribedinTable1.

Adefinitionofresearchthatcanbededucedfromthereference modelis:‘ResearchisaConsciousActionthataimsforNew Knowl-edge,emanatesfromoneorseveralQuestionsatHand,studiesone orseveralContexts,buildsuponExistingKnowledge,usesoneor sev-eralScientificMethods,isdocumentedinoneDescribedProcedure, requiresTransparencyandrelatestooneorseveralSystemsofRules’.

3.2. Phase2:Concepthierarchyofresearchqualityandrelated definedterms

Asasecondphase,theconceptmodelofresearchwasusedas abasisfordetermininganddiscussingdifferentaspectsofquality andevaluationinresearch.Withthisprocedure,important qual-ityaspectsthatcouldberelatedtotheconceptofresearchwere defined.

(6)

Fig.4. Conceptmodelofresearch.

Fourmainaspectsemergedfromtheworkshopsheldaspartof thisproject,andtheyarelargelyconsistentwithearlierresearch. Previousresearchhaspointedout thefirstthree aspects, credi-ble,contributoryandcommunicable(Mårtensson,2003;Mårtensson andMårtensson,2007).Duringtheworkshopsinthisproject, how-ever,itwasfoundthatthesethreeaspectsdidnotcaptureallthose thatemergedinthediscussions.Throughthemodellingapproach a fourth aspectemerged, conforming. By conforming, we mean researchthatisalignedwithregulations,ethicalandsustainable.

The concept hierarchy defined during the workshops thus breaksdownintofourmainareaslabelled‘Credible’,‘Contributory’, ‘Communicable’and‘Conforming’(Fig.5).These branches consti-tute themain dimensions of research quality according toour concept hierarchy, or quality model. Each of these wasfurther specifieduntil32conceptswereestablished,including9branch

conceptsand23leafconcepts.Formaldefinitionswerethengiven toallconceptsinthequalitymodel,andallconceptsinthe ref-erencemodelfromthefirstphasewereneededfordefiningthese concepts.

FortheCrediblepartoftheconcepthierarchy,sixrelated con-ceptsweredeveloped(Table2).Itwaspossibletoformulateformal definitionsbasedonconceptsinthereferencemodelfromthefirst phase.Forexample,Reliablewasdefinedas:‘ThechosenScientific MethodisappropriateforthepresentQuestionatHandandContext, andisdocumentedinaDescribedProcedurethatotherscoulduse toreachasimilarresultinthesameContext’.

FortheContributorypartoftheconcepthierarchy,sevenrelated conceptsweredeveloped(Table3),andformaldefinitionswere setupaccordingly.Forexample,CurrentIdeawasdefinedas:‘The QuestionatHandisinaccordancewiththeContext’.

Table1

Definitionsofallconceptsintheconceptmodelofresearch.

Term Definition

Actor APersoninitiatingand/orperformingaConsciousAction Beneficial ApositiveeffectofNewKnowledgeforaTargetGroup Consciousaction Aprocessinitiatedand/orperformedbyanActor.

Context AnenvironmentalorintellectualsettingwheretheResearchtakesplaceand/orisstudied,andwhereExistingKnowledgeisvalid Describedprocedure AdescriptionofhowtheResearchwillbeperformedanddocumentedaccordingtotheRulesforDescription

Existingknowledge KnowledgethatisbuiltonbytheResearch,existsinaContext,canbedocumentedinaSourceandisexpandedwithNewKnowledge Newknowledge KnowledgethatexpandsExistingKnowledge,isProvable,andisBeneficialforaTargetGroup

Person Ahumanbeing

Provable EvidencethattheNewKnowledgeisdemonstrable Questionathand AresearchquestionthatisthebaseforResearch

Relationship ArelationbetweentwoConsciousActionsshowinghowthoseactionsinteract

Research AConsciousActionthataimsforNewKnowledge,emanatesfromoneorseveralQuestionsatHand,studiesoneorseveralContexts, buildsuponExistingKnowledge,usesoneorseveralScientificMethods,isdocumentedinoneDescribedProcedure,requires TransparencyandrelatestooneorseveralSystemsofRules

Rulesfordescription RulesdescribingwhataDescribedProcedureshouldcontain,includingitsintentionsandresults.ThiscandifferinregardstoContext, ScientificMethod,SystemofRules,ExistingKnowledgeandQuestionatHand

Scientificmethod AdescribedandprecisetechniqueusedforconductingtheResearch Source Documents,databasesorothermediathatcontainExistingKnowledge

Systemofrules Legalrequirements,regulations,normsandotherguidelinesthatinfluencehowResearchshouldbeperformed Targetgroup Individuals,organizations,enterprisesand/orsocietythatbenefitfromNewKnowledge

(7)

Fig.5.Concepthierarchyofresearchquality.

Table2

DefinitionsofconceptsrelatedtoCredible.

Term Definition

Credible ResearchthatisCoherent,Consistent,Rigorousand Transparent

Rigorous ResearchthatisContextual,InternallyValidand Reliable

Internallyvalid AcorrectScientificMethod(incl.researchdesign)is usedinrelationtotheQuestionatHandandContext, andNewKnowledgeisProvable

Reliable ThechosenScientificMethodisappropriateforthe presentQuestionatHandandContext,andis documentedinaDescribedProcedurethatothers couldusetoreachasimilarresultinthesameContext Contextual ExistingKnowledgethatisrelevantfortheContextis

used,andispresentedaccordingtoRulesfor Description

Consistent NewKnowledgeislogicallylinkedtoExisting KnowledgeandisinaccordancewiththeScientific MethodandQuestionatHand

Coherent AdequateconsiderationisgiventoExistingKnowledge inthechosenContext

Transparent RelevantNewKnowledgeinthereportingofresearch resultsisincludedandtheprocessisdescribedin relationtotheQuestionatHand,ScientificMethodand ExistingKnowledge

FortheCommunicablepartoftheconcepthierarchy,five con-cepts, with formal definitions, were developed (Table 4). For example,Readablewasdefinedas:‘CorrectlanguageintheResearch DocumentationfortheTargetGroup’.

FortheConformingpartoftheconcepthierarchy,fiverelated concepts,withformaldefinitions,weredeveloped(Table5).For example,Morally Justifiablewasdefined as: ‘TheResearch com-plieswithcurrentlyapplicableethicalstandardsasdescribedin the System of Rules’. All concepts were, as in the other parts, textuallydefined byusing conceptsfromtheconceptmodel of research.

Table3

DefinitionsofconceptsrelatedtoContributory.

Term Definition

Contributory ResearchthatisOriginal,RelevantandGeneralizable Original ResearchthathasanOriginalIdea,usesanOriginal

ProcedureandproducesanOriginalResult Originalidea TheQuestionatHandhasnotbeenaskedbeforein

thecurrentContextorisinterpretedinanovelway Originalprocedure DescribedProcedureisoriginalinrelationtothe

QuestionatHand

Originalresult NewKnowledgeisProvableinrelationtoExisting Knowledge

Relevant ResearchthathasaRelevantResearchIdea, ApplicableResultandCurrentIdea

Relevantresearchidea QuestionatHandisrelevantforthecurrentTarget Group

Applicableresult NewknowledgeisBeneficialforthecurrentTarget Group

Currentidea TheQuestionatHandisinaccordancewiththe currentContext

Generalizable NewKnowledgeispracticallyortheoreticallyuseful inContextsotherthantheonestudied

Table4

DefinitionsofconceptsrelatedtoCommunicable.

Term Definition

Communicable ResearchthatisConsumable,Accessibleand Searchable

Consumable ResearchthatisStructured,Understandableand Readable

Structured TheResearchdocumentationfollowstheRulesfor Description

Understandable ThelanguageintheResearchdocumentationis understandablefortheTargetGroup

Readable CorrectlanguageintheResearchdocumentationfor theTargetGroup

Accessible NewKnowledgeiseasilyavailabletotheTargetGroup Searchable ThedocumentedNewKnowledgeisstructured

accordingtotheRulesforDescriptionandeasilyfound bytheTargetGroup

(8)

Table5

DefinitionsofconceptsrelatedtoConforming.

Term Definition

Conforming ResearchthatisRegulatoryAligned,Ethical andSustainable

Alignedwithregulations TheResearchcomplieswithcurrently applicablelegalaspectsoftheSystemofRules Ethical TheResearchisMorallyJustifiable,Openand

supportsEqualOpportunities Morallyjustifiable TheResearchcomplieswithcurrently

applicableethicalstandardsasdescribedinthe SystemofRules

Open TheResearchdemonstratesTransparencywith currentlyapplicableethicalstandardsas describedintheSystemofRules Equalopportunities TheResearchisconsistentwithequal

treatmentaccordingtotheSystemofRules Sustainable TheResearchcomplieswithsustainable

developmentaspectsasdescribedinthe SystemofRules

4. Discussion 4.1. Mainfindings

Thisstudy presents a general,generic and multidisciplinary frameworkfor identifyingwhat thequalityofresearchpractice mightbe,andhowthatmodelmightbeusedtoevaluateresearch quality.First,wecreatedaconceptmodel ofresearchasa phe-nomenon,ourreferencemodel.Thismodelincludedatotalof18 definedconceptsandtheirrelationships.Second,wecreateda con-cepthierarchy,ourqualitymodel,with32conceptswithadefined terminology.Based onthisconcepthierarchy,thedifferent ele-mentscanbeusedeither withequalweights allocatedtoeach concept,orbyallocatingdifferentweightsaccordingtopreferences inthespecificcontextforevaluatingresearchquality.Thismeans thatintheapplicationofourcomprehensivequalitymodelsome aspectsmayevenbeallocatedtheweightofzero,creatingaspecial caseofourmodelwherecertainaspectsarenotbeincludedinthat specificcontextand/orsituation.Thepurposeofourqualitymodel istoofferabroadcomprehensivemodel,encompassingothermore specificcontextualsituations.

Ourintentioninthisstudyhasbeentoformulateaframework fortheassessmentofthequalityofresearchpractice.Weargue thatthisisausefulapproachfordiscussingresearchpracticeand itsqualityfrommanyperspectives,andcanhelptoadvance dis-cussions onresearchquality within and acrossdisciplines.Our intention,furthermore,isthatthisgeneralframeworkcanbeused asaplatformfordevelopingsubject-specificversionsofthe frame-work,whichcanbefittedtoacontext.

4.2. Meaningoftheresultsandcomparisonwiththeliterature Ourfindingsregardingresearchasaphenomenoncannotbe eas-ilycomparedwithpreviousresearchas,toourknowledge,thisarea hassofarreceivedlimitedmodellingattention.Theconceptsinthe conceptmodelofresearchweredefinedusingintrinsicdefinitions, i.e.definitions basedonrelations tootherconcepts.These self-consistentdefinitionsareconsideredessentialinordertoachieve highreliabilitywhenusingthemodel.

Ourfindingsinthesecondphase,relatedtotheCredibleconcept, canreadilybecomparedwithpreviousworkbyShipman(1982) andGummesson(1991)thatsuggestsandintroducestheconceptof Transparency.However,itisimportanttorealizethatTransparency maybemoreorlesspronouncedincertainresearchpractices.For example,researchperformedata universityis oftenveryopen andtransparent,whileresearchperformedwithinacommercial companymaynotbe.Botharestillexamplesofresearchandthey

maybothbeofhighquality,butsomewheretheremightbealine betweenresearchleadingto‘proprietaryassets’,andacompany’s researchactivitiesthatresultinpublicationsandthusappearmore openandtransparent.SeealsothediscussioninSection4.5.

Inaddition,RubinandRubin(1995)emphasizeConsistencyand Coherenceinbothquantitativeandqualitativeresearch,whichis inlinewithourmodel.Lahtinenetal.(2005)alsoinclude Coher-enceasanimportantpartofscientificquality.Ourmodelissimilar totheworkofMårtenssonandMårtensson(2007)regardingthe introductionoftheconceptCredible;however,theconceptCoherent is added in ourmodel. Internal Validityand Reliability are con-ceptsthathavebeendiscussedfurtherby,forexample,Silverman (1993),aswellasbyRubinandRubin(1995).AlborzandMcNally (2004)alsostresstheimportanceofRigourwhenreviewing evi-denceregardingaccesstohealthcare.Keen(1991)and Maxwell (1996)haveemphasizedtheimportanceoffurtherperspectives forContextual,suchasspecifyingandanalysingaconceptual con-text,which is also in linewith thework of Aminand Roberts (2008).

TheContributorydimensioncanbecomparedtopreviouswork byMårtenssonandMårtensson(2007),wheretheconceptof Con-tributoryisused.OurconceptofRelevantResearchIdeais,however, somewhatdifferent.LikeBerthonetal.(2002),wefurther devel-opedOriginalbyusingthreeunderlyingconcepts.Incontrastto themodelbyBenbasatandZmud(1999),Accessibleinourmodel isnotapartofRelevance.Inasomewhatdifferentway,Rosemann andVessey(2008)divideRelevanceintothreedifferentdimensions (Importance,AccessibilityandSuitability).Further,AugierandMarch (2007)haveexploredRelevanceandseveralaspectsofusefulness inamoredetailedway.AlborzandMcNally(2004)alsoemphasize theimportanceofRelevancewhenreviewingresearchonaccessto healthcare.Inaddition,Vermeulen(2005)arguesthat synthesiz-ingRelevanceandRigourrequiresasystemicchangewhentryingto tacklearealpracticalquestionandsearchingforanswersina rigor-ousway.InlinewithMason(1996)andLeeandBaskerville(2003), wefoundGeneralizabilitytobebothpracticallyandtheoretically relevant.

In the Communicableconcept, we furtherexplored concepts usedbyMårtenssonandMårtensson(2007)byaddingthree sub-groupstoConsumable,which hasbeenarguedtobeofprimary importancebyRobeyandMarkus(1998)andDesouzaetal.(2006). ThesethreesubgroupsareStructured,UnderstandableandReadable, andtheyallconcernaspectsoftheresearchDocumentation.Inthe Communicableconcept,theothertwoconceptsareAccessibleand Searchable.Boththeseconceptsconcernhowtheresearch docu-mentationcanbeusedbythepotentialaudience,orTargetGroup. Itcanbearguedthatthispartofthemodelismorelikeameans toanendthananaspectofresearchitself.Atthesametime,itis verylikelythatitwillbedifficultforresearchthatisnot consum-able,accessibleandsearchable,ornotcommunicable,toreachthe targetgroup,andthusthevalueoftheresearchcanbequestioned. OurfindingsonConformingaresomewhatnewcomparedto pre-viousconceptualexplorativeresearch.However,Pickersgill(2012) recentlyarguedthatsciencetodayisanethicalbusiness,that sci-enceandethicsareinmanywaysco-produced,andthatscienceand technologyscholarshavealonghistoryofdevelopingand imple-mentingbioethicsinpractice.InabroadsensethisimpliesaMoral andOpenorientationthatalsotakesintoaccountEqual Opportu-nitiesandiswellalignedwiththephilosophicalunderpinningsof academiaasa(rational)meritocracy.TheconceptsAlignedwith Reg-ulationsandSustainablearerarelymentionedinpreviousresearch, eventhoughbothshouldbeconsideredasessentialinallresearch practice.Theimportanceoftheconformingaspectsofresearchis highlightedbytheintensiveon-goingdebateonplagiarism, uneth-icalbehaviourandnorms,notonlyamongstudents,butalsoamong academics(e.g.HonigandBedi,2012).

(9)

4.3. Researchversusdevelopment

In industrial settings for research, the terms ‘research’ and ‘development’ (R &D) are oftenused together. This raisesthe questionofwhetherthereis adifferencebetweenthepractices ofresearchanddevelopment.Onecouldarguethatresearchand developmentare closelyrelated, which can bethecase, asthe developmentofaproductorservice isoftenbasedonprevious research.However,developmentdoesnot needtobebased on research,norisresearchalwayslinkedtothedevelopmentof some-thing.Thisisalsodiscussedintheliterature,where,forexample, Barge-GilandLópez(2014)clearlyseparateresearchfrom devel-opment.DiasandBresciani(2006)alsoseparateresearchactivities from developmentactivities within a company, indicating that thesemightberelated,butdonotneedtobe.Thebulkof devel-opmentproceduresperformedtodayareprobablyratherfarfrom research,inthesensethattheaimofdevelopmentisusually a (sellable)productorservice,buttheaimofresearchis(only)new knowledge.Therefore,researchperformedwithinacompanymay beseenassuchwhenithassomedegreeofopennessand trans-parency (such as when it is published) and is not only aimed atdevelopingaproductorservice.Therearealsolinksbetween companiesanduniversitiesconcerningdevelopmentand innova-tionactivities.Previousresearchhasshownthatdifferentsearch strategiesadoptedbycompaniesinfluencetheuseofuniversity knowledgeindevelopment(LaursenandSalter,2004).The inten-tionofourframework,however,shouldbeseenastodefine(and lateron,toevaluate)researchpracticeonly,andnotdevelopment practice.

4.4. Anexampleasanillustrationofthemodel

If,then,weweregoingtoapplyourmodelofthecriteriaforhigh qualityresearchpractice,whatwouldthislooklike?Letusimagine alargeuniversitythatwantstoevaluatethequalityofitsresearch. Usingourapproach,averysimplifiedversionoftheprocesswould looksomethinglikethis:

(1)Selectionof anevaluationcommittee(preferably multidisci-plinary,primarilyforlegitimacyreasons).

(2)Discussion within the committee regarding the four over-arching concepts, credible, contributory, communicable and conforming,theunderlyingsub-concepts,andwhatthesemean forthemembers.

(3)Weighting of concepts, identifyingwhat theparticular uni-versity believes to be the most important concepts or sub-concepts.Forexample,thecommitteemightagreethatthe researchattheuniversityshouldreachouttotheworldina bet-terwaythanbefore,butthatitdoesnotneedtobegeneralizable tootherdomains.Inthiscase,adecisionmaybemadetogive Communicableanditssub-conceptsaweightof1.5andallthe otherconceptsaweightof1.0,exceptthatGeneralizablewould receiveaweightof0.5.

(4)Operationalizationofconcepts;collectionandanalysisofdata onresearchgroups’practicesorprojects;andanalysisofthe resultsbasedontheweightsofthedifferentconcepts.Thiswill probablycreatea fertilebasefora verylivelydiscussionon themeasures,weightsandendresults.Thetypeofdatathat shouldbecollectedwouldbecloselyrelatedtotheconcepts thathavebeenassignedthehighestweights.IfCommunicable hasbeenprioritized,thenprobablypublicationdataofvarious typesshouldbecollectedandscrutinizedintermsof,for exam-ple,numberofpublications,impactfactors,citationsetc.Butif Crediblehasbeengiventhehighestweight,itcouldbecome importanttoanalysedataonappliedresearchmethods.

4.5. Strengthsandlimitations

Itisimportanttounderstandthattheaimofthisstudyisto describewhatconstitutesresearch,andthentousethat knowl-edge to describe how research quality might be measured. To dothis,anumberofdifferentapproachescouldhavebeenused, includingbothqualitativeandquantitativemethods.Weuseda conceptmodellingapproach,assumingthatthesocialrealmmay besubjectedtothesamescientificmethodsofinvestigationand empiricismasthenaturalworld.Ourviewcouldperhapsbeseen asDurkheim(1898/1982),aswetrytoconstructconceptsinthe abstractinordertoformworkablecategoriesforempiricalanalysis. Wecouldhaveconsideredseveralqualitativeapproachesother thanconceptmodelling,suchasadatasamplebasedonastructured questionnaire,anopenquestionnaire,orafocusgroupinterview, followedbycontentanalysisandthendeterminationofthe var-iouscriteriaorconceptsthatcouldbederived fromthecontent analysis.However,webelievethat,withourliteraturereview,the opendiscussionsduringtheprocess,andthebroadrepresentation oftheresearchersintheteam,wehaveappliedamethodologically rigorousapproach.

Amorequantitativeapproachcould,forexample,haveincluded interviews,questionnairesfollowedbyconsensusmethodssuchas theDelphimethod,ethnographicfieldwork,ordiscourseanalysis. However,webelievethataquantitativeapproachwouldhavebeen lessappropriateatthisstage,becausetheconceptshadnotactually beendefined,butthatsuchanapproachcouldverywellbeusedin afollow-upstudybasedonourmodel.

The study was carried out by a multidisciplinary group of authors,whohavebeeninvolvedinqualityassessmentsandthe evaluation of researchwithin medicine,life sciences, computer andsystemsciences,socialsciences,businessadministration,and economics.Theyhave experiencefromquality projectsat their respectiveuniversitiesandatotheruniversities(seeSection2.1for moredetail),allofwhichusedverydifferentdimensionsand vari-ablesintheirapproachestoresearchevaluation.Insomerespects ourworkcannotbegeneralized,asitisinitselfbasedonresults fromsocialconstructivistandqualitativeactivities.Incomparison torelatedstudies,however,itisbasedonamorerigorousmethod andapproach.

Weusedamultidisciplinaryapproachwhenmodelling,inline withouraims,whilebeingwellawarethatthisisa fundamen-tallydifferentwayofworkingfromtheapproachwhenconducting single-disciplinework.Unexpectedly,thiswasanobstacleto mod-elling at a more detailed level, as representatives of different disciplinessometimeshelddifferentviews.However,thebroad representationofresearchersfromseveraldifferentacademicareas isanimportantstrengthinrelationtothecredibilityofthemodel anditsfutureapplicability.Thebroadrepresentationofthe dif-ferentdomainexpertswhoparticipatedinthisstudyisthusan advantage,andtothebestofourknowledgeitdoesnotfeaturein relatedwork.However,weareawarethatevenifourexperience fromvariousacademicfieldsis broad,wecannotclaimthatwe coverallpossibleaspectsordisciplines.Specialistsinphilosophy orastronomy,forexample,mighthavereacheddifferentresults, although weargue thatourmodel would alsobevalidin such domains.Infutureresearch,however,thiscurrentstudycouldbe builtuponbyapplyingandtestingthegeneralandcomprehensive modelpresentedhereindifferentdisciplinarycontexts.

Concept modelling is a time-consuming process, and when experts from different domains are included, an initial learn-ing phase is necessary to reach a common ‘language’ and an understanding of themethod. Of course,concept modellingas suchcouldbediscussed,notonly inrelationtowhetherornot it is a good method to use in a context like this, but also in relation towhether thedefined conceptsare well-defined and

(10)

understandable.Asdescribedearlier,researchcanbeevaluatedin amultitudeofdimensions,oftenwithacertainoverlap.Further, thereisfrequentlyaneedforaformalconceptmodelandan estab-lishedterminologyinordertosupporttheevaluationofacertain topic.Variousapproachesinrelatedworkarecurrentlyconverging towardstheuseofspecifiedterms,basedonanunderlying for-malconceptmodel.Forexample,theSNOMEDCTmodeliswidely usedwithinhealthcare(SNOMEDCT,2014).Thereisconsequentlya needtoinvestigatetheconceptualcontentofthemultidisciplinary notionofresearchandrelatedqualityandevaluationissues. How-ever,developingatrulyconcept-orientedterminologyandmodel isademandingendeavour.

4.6. Implicationsforpolicymakers,researchersandfuture research

Ourmodelofresearchcanbeusefulinsituationsinwhichthere isaneedtodifferentiateresearchfromotherhumanactivitieswith socialaims—suchasreligion,artandjournalism.Forexample,when practicessuchashomeopathyhavebecomewidespread,and vari-ousdietsarefrequentlypromotedindifferentmedia,ourmodelcan beusedtocreateacleardistinctionbetweenclaimsbasedon gen-uinescientificstudiesandclaimsbasedonlyonbeliefsortraditions. Themodelcanbeseenandusedbothasatypeofchecklist(‘are someaspectsneglected?‘)andasapossiblefundamental struc-turefordiscussingthequalityofresearch.Theintentionwasfor thisgeneralcomprehensiveframeworktoformaplatformforthe furtherdevelopmentofcommonconcepts,termsandcriteriafor evaluationsofresearchqualitywithinandacrossspecificdomains, andtherebytocontributetoeffortstoimproveresearchqualityand understanding.

Furthermore,ourmodelof researchqualitycanbeuseful in severalacademicsituations.First, whenplanning,writing grant applicationsanddesigningresearch,theframeworkprovidesaset ofusefuldimensionstocoverandtoconsider.Second,the dimen-sionscanbeusefulinthedifferentadditionalstepsoftheresearch processinspecificprojects,forexampleindataanalysis.Third,it mayalsobeusedasaguidingprincipleindifferentevaluation pro-cessessuchasreviewprocesses.Fourthly,inmoregeneralquality workandtheassessmentorevaluationofconstellationsor insti-tutions,itmaybeofuseinaplethoraofsettings.Fifth,themodel canbeusedindoctoralprogrammesinwhich PhDstudentsare educatedandtrainedtoevaluatetheirownandothers’research activities.

Itisourhopeandcontentionthatthemodelpresentedhere maypromote understandingof thegeneric dimensionsof qual-ityinresearchpractice.Severalmorespecificapplicationsofour modelcouldcallforfurtherdevelopmentandadjustment,andalso forweightsfordifferentconceptsorpartsofourspecificsystem. Onecouldargue,forexample,thatitisverylikelythatresearchin medicinewoulduseweightsforsomeparametersthatwere dif-ferentfromtheweightsusedforresearchinlinguistics,computer scienceorsocialsciencestudies.Futurestudiesmaydevelopmore discipline-specificconceptmodelswithspecificweightsforeach concept.Furthermore,anapproachtotestinginter-evaluator reli-abilityinarealisticsituation,asintheCanadianstudymentioned above,would bedesirablein ordertocomeclosertoa science-orientedapproach to evaluating researchpractice (Kmet et al., 2004).Ofcoursethemodelproposedinthispaperdoesnotdelivera completedelineationofallthoughts,wordsandfactorsofrelevance toresearchpracticequality.Forexample,theconceptCurrentIdea couldbefurtherspecifiedandmodelledindifferentprofessions.

In sum, our model is expected to help us to represent and understandthecriteriaforhighqualityresearchinamore con-sistentmanner. It is tobeused both in itscurrent format and alsofor furtherdevelopmentof concepts,terms and criteria in

specificdomains.Wehopethatitwilltherebycontributetofuture effortstoimproveeveryday academicwork,linkingresearchers anddeveloping researchpractice,asassessmentand evaluation arefundamentalandincreasinglyimportantaspectsofallresearch practice.

5. Concludingremarks

Fromamultidisciplinaryperspective,acomprehensivequality modelincluding32concepts,basedonfourmainareas(Credible, Contributory,CommunicableandConforming)wasdevelopedfor describingthemaincontentofthequalityofresearchpractice.A modelofresearchasa phenomenon,witha totalof 18defined conceptsandtheirrelationships,wasneededtodefinethe qual-ityconcepts.Thequalitymodelmaybeusedasthestructurefor thefurtherdevelopmentofrelevantelements,weightsand opera-tionalizationsrelatedtothequalityofresearchpracticeindifferent academicfields.

Authors’contributions

GHNandPMconceivedanddesignedthestudy.SBWled,and alltheauthorsparticipatedin,themodellingwork.Alltheauthors wereinvolvedinwritingthemanuscript,andallofthemreadand approvedthefinalmanuscript.

Acknowledgements

WewishtothanktheStockholmSchoolofEconomicsfor sup-portduringthemeetingsandworkshops.

References

Alborz,A.,McNally,R.,2004.Developingmethodsforsystematicreviewingin healthservicesdeliveryandorganisation:anexamplefromareviewofaccess tohealthcareforpeoplewithlearningdisabilities.Part2.Evaluationofthe literature—apracticalguide.HealthInf.Lib.J.21,227–236.

Amin,A.,Roberts,J.,2008.Knowinginaction:beyondcommunitiesofpractice. Res.Policy37,353–369.

Augier,M.E.,March,J.G.,2007.Thepursuitofrelevanceinmanagementeducation. Calif.Manage.Rev.49,129–146.

Barge-Gil,A.,López,A.,2014.RversusD:estimatingthedifferentiatedeffectof researchanddevelopmentoninnovationresults.Ind.Corp.Change,1–37. Benbasat,I.,Zmud,R.W.,1999.Empiricalresearchininformationsystems:the

practiceofrelevance.MISQ.23,3–16.

Berezow,A.B.,Hartsfield,T.,2012.Whatseparatessciencefromnon-science?Real ClearSci.,2012/05/30.Availableatwww.realclearscience.com.

Berthon,P.,Pitt,L.,Ewing,M.,Carr,C.,2002.PotentialresearchspaceinMIS:a frameworkforenvisioningandevaluatingresearchreplication,extensionand generation.Inf.Syst.Res.13,416–427.

Boman,M.,BubenkoJr.,J.A.,Johannesson,P.,Wangler,B.,1997.Conceptual Modelling.UpperSaddleRiver,NJ,PrenticeHall.

CIVR(CommitteeforEvaluationofItalianResearch),2006.GuidelinesforResearch Evaluation,Availableat

http://vtr2006.cineca.it/documenti/lineeguidaEN.pdf.

Dash,D.P.,2009.Scienceasreflectivepractice:areviewofFrederickGrinnell’s book,everydaypracticeofscience.J.Res.Pract.5(ArticleR1).

Desouza,K.,ElSawy,O.,Galliers,R.,Loebbecke,C.,Watson,R.,2006.Beyondrigor andrelevancetowardsresponsibilityandreverberation:informationsystems researchthatreallymatters.Commun.AIS17,341–353.

Dias,R.T.,Bresciani,S.,2006.R&Dandknowledge:atheoreticalassessmentofthe internationalisationstrategies.Int.J.Technol.PolicyManage.6,1–32. Dubé,L.,Paré,G.,2003.Rigorininformationsystemspositivistcaseresearch:

currentpractices,trends,andrecommendations.MISQ.27,597–635. Durkheim,E.,1898/1982.TheRulesofSociologicalMethod.TheFreePress,New

York,NY.

EuropeanScienceFoundation,2012.Evaluationinresearchandresearchfunding organisations:Europeanpractices.In:AreportbytheESFMember OrganisationForumonEvaluationofPubliclyFundedResearch.,ISBN 978-2-918428-83-1.

ExternalResearchAssessment(ERA),2010.ExternalResearchAssessment. KarolinskaInstitutet,Stockholm,Availableatwww.ki.se.

Gall,M.D.,Borg,W.R.,Gall,J.P.,1996.EducationalResearch:AnIntroduction,sixth ed.Longman,WhitePlains,NY.

(11)

Gibbons,M.,Limoges,C.,Nowotny,H.,Schwartzman,S.,Scott,P.,Trow,M.,1994. TheNewProductionofKnowledge:TheDynamicsofScienceandResearchin ContemporarySocieties.SAGE,London.

Grinnell,F.,1990.Endingsofclinicalresearchprotocols:distinguishingtherapy fromresearch.EthicsHum.Res.12,1–3.

Grinnell,F.,2000.Thepracticeofscienceattheedgeofknowledge.ChronicleRev., March24.Availableathttp://chronicle.com.

Grinnell,F.,2009.EverydayPracticeofScience:WhereIntuitionandPassionMeet ObjectivityandLogic.OxfordUniversityPress,NewYork,NY.

Gummesson,E.,1991.QualitativeMethodsinManagementResearch.Sage Publications,NewburyPark,CA.

Hedin,A.,Jernberg,L.,Lennér,H.C.,Lundmark,T.,Wallin,S.-B.,2007.ToMeanand MeasureistheSameThing(Attmenaochmätasammasak).Studentlitteratur, Stockholm.

Hessels,L.K.,vanLente,H.,2008.Re-thinkingnewknowledgeproduction:a literaturereviewandaresearchagenda.Res.Policy37,740–760. Hicks,D.,Wounters,P.,Waltman,L.,deRijcke,S.,Rafols,I.,2015.TheLeiden

Manifestoforresearchmetrics.Nature520,429–431.

Honig,B.,Bedi,A.,2012.Thefoxinthehenhouse:acriticalexaminationof plagiarismamongmembersoftheAcademyofManagement.Acad.Manage. Learn.Educ.11,101–123.

ISO/TR24156:2008,2008.GuidelinesforUsingUMLNotationinTerminology Work,Availableathttp://www.iso.org.

ISO/IEC/IEEE42010:2011,2011.SystemsandSoftwareEngineering—Architecture Description,Availableathttp://www.iso.org.

Israel,G.,2005.Thescienceofcomplexity:epistemologicalproblemsand perspectives.Sci.Context18,479–509.

Jiménez-Contreras,E.,deMoyaAnegón,F.,DelgadoLópez-Cózar,E.,2003.The evaluationofresearchactivityinSpa.TheimpactoftheNationalCommission fortheEvaluationofResearchActivity(CNEAI).Res.Policy32,123– 142.

Keen,P.G.W.,1991.Relevanceandrigorininformationsystemsresearch: improvingquality,confidence,cohesionandimpact.In:Klein,H.K.,Nissen, H-E.,Hirschheim,R.(Eds.),InformationSystemsResearch:Contemporary ApproachesandEmergentTraditions.IFIPElsevierScience,Philadelphia,PA, pp.27–49.

Klein,H.K.,Myers,M.D.,1999.Asetofprinciplesforconductingandevaluating interpretivefieldstudiesininformationsystems.MISQ.23,67–94. Kmet,L.M.,Lee,R.C.,Cook,L.S.,2004.Standardqualityassessmentcriteriafor

evaluatingprimaryresearchpapersfromavarietyoffields.In:HTAInitiative #13.AlbertaHeritageFoundationforMedicalResearch(AHFMR),Edmonton, AB,Availableathttp://www.ihe.ca.

Lahtinen,E.,Koskinen-Ollonqvist,P.,Rouvinen-Wilenius,P.,Tuominen,P., Mittelmark,M.B.,2005.Thedevelopmentofqualitycriteriaforresearch:a Finnishapproach.HealthPromot.Int.20,306–315.

Laursen,K.,Salter,A.,2004.Searchinghighandlow:whattypesoffirmsuse universitiesasasourceofinnovation?Res.Policy33,1201–1215. Lave,J.,Wenger,E.,1991.SituatedLearning:LegitimatePeripheralParticipation.

CambridgeUniversityPress,Cambridge.

Lee,A.S.,Baskerville,R.L.,2003.Generalizinggeneralizabilityininformation systemsresearch.Inf.Syst.Res.14,221–243.

Mårtensson,A.,Mårtensson,P.,2007.Extendingrigorandrelevance:towards credible,contributoryandcommunicableresearch.In:In15thEuropean ConferenceonInformationSystems(ECIS2007),StGallen,Switzerland. Mårtensson,A.,2003.ManagingMission-CriticalITintheFinancialIndustry.The

EconomicResearchInstitute,StockholmSchoolofEconomics,Stockholm. Mason,J.,1996.QualitativeResearching.SagePublications,London.

Mathison,S.(Ed.),2005.EncyclopediaofEvaluation.SagePublications,Thousand Oaks,CA.

Maxwell,J.A.,1996.QualitativeResearchDesign:AnInteractiveApproach.Sage Publications,ThousandOaks,CA.

Mertens,D.M.,2015.ResearchandEvaluationinEducationandPsychology: IntegratingDiversitywithQuantitative,Qualitative,andMixedMethods.Sage Publications,ThousandOaks,CA.

Nickelsen,N.C.,2009.Rethinkinginterventionistresearch:Navigatingoppositional networksinaDanishhospital.J.Res.Pract.5(ArticleM4).

NIH,2008.NationalInstitutesofHealthannouncesupdatedcriteriaforevaluating researchgrantapplications.In:NoticeNumber:NOT-OD-05-002,Availableat http://www.nih.gov.

Nuopponen,A.,1994.ConceptSystemsforTerminologicalAnalysis (Begreppsystemförterminologiskanalys).UniversityofVaasa,Vaasa. Patton,M.Q.,1990.QualitativeEvaluationandResearchMethods,seconded.Sage,

NewburyPark,CA.

Pickersgill,M.,2012.Theco-productionofscience,ethicsandemotion.Sci. Technol.Hum.Values37,579–603.

Quaye,S.J.,2007.Voiceoftheresearcher:extendingthelimitsofwhatcountsas research.J.Res.Pract.3(ArticleM3).

Robey,D.,Markus,M.L.,1998.Beyondrigorandrelevance:producingconsumable researchaboutinformationsystems.Inf.Resour.Manage.J.11,7–15. Rosemann,M.,Vessey,I.,2008.Towardimprovingtherelevanceofinformation

systemsresearchtopractice:theroleofapplicabilitychecks.MISQ.32,1–22. Rubin,H.J.,Rubin,I.S.,1995.QualitativeInterviewing:TheArtofHearingData.

SagePublications,ThousandOaks,CA.

ScienceCouncil,2009.Availableat:http://www.sciencecouncil.org.

Shewchuk,R.M.,O’Connor,S.J.,Williams,E.S.,Savage,G.T.,2006.Beyondrankings: Usingcognitivemappingtounderstandwhathealthcarejournalsrepresent. Soc.Sci.Med.62,1192–1204.

Shipman,M.,1982.TheLimitationsofSocialResearch.Longman,London. Silverman,D.,1993.InterpretingQualitativeData:MethodsforAnalysingTalk,

TextandInteraction.SagePublications,London.

SNOMEDCT,2014.Availableat:http://www.ihtsdo.org/snomed-ct/snomed-ct0/. Sutherland,H.J.,Meslin,E.M.,daCunha,R.,Till,J.E.,1993.Judgingclinicalresearch

questions:whatcriteriaareused?Soc.Sci.Med.37,1427–1430.

Tsao,J.Y.,Boyack,K.W.,Coltrin,M.E.,Turnley,J.G.,Gauster,W.B.,2008.Galileo’s stream:aframeworkforunderstandingknowledgeproduction.Res.Policy37, 330–352.

Ulrich,W.,2006.Rethinkingcriticallyreflectiveresearchpractice:beyondPopper’s criticalrationalism.J.Res.Pract.2(ArticleP1).

UnifiedModelingLanguage(UnifiedModelingLanguageTM(UML®).Availableat

http://www.uml.org.

Vermeulen,F.,2005.Onrigorandrelevance:fosteringdialecticprogressionin managementresearch.Acad.Manage.J.48,978–982.

Figure

Updating...

Related subjects :