Kitchenham et al. report 53 unique systematic literature reviews in software engineering being published between 2004 and 2008 [103,104]. They conclude that there is a growth of the number of systematic literature reviews being published, and that the quality of the reviews tend to be increasing too. However, still there is large variation between those who are aware of and use any systematic guidelines for its conduct, and those who are not referring to any guidelines.
In one of those systematic literature reviews, Sjøberg et al. [161] survey the experimental studies conducted in software engineering. They searched nine journals and three conference proceedings in the decade from 1993 to 2002, scanning through 5,453 articles to identify 103 experiments, i.e. 1.9% of the papers presented experiments. The two most frequently research categories are Software life-cycle/engineering (49%) and Methods/Techniques (32%) classified according to Glass et al’s scheme [63]. This is due to the relatively large number of experiments on inspection techniques and object-oriented design techniques, respectively.
Using the same set of primary studies, Dyb˚a et al. [49] reviewed the statistical power in software engineering experiments, and Hannay et al. [72] reviewed the use of theory in software engineering. Dieste et al. [43] investigated different search strategies on the same set of studies, whether titles, abstracts or full texts should be searched, and also aspects related to which databases to search.
Early attempts at synthesizing five experiment on inspection techniques by Hayes [74] and Miller [121] indicate that the software engineering experiments in this field are not sufficiently homogenous to allow for application of statistical
Table4.1Differencebetweenmappingstudiesandsystematicliteraturereviews,accordingtoKitchenhametal.[106] SLRelementsSystemicmappingstudySystematicliteraturereview GoalsClassificationandthematicanalysisofliteratureonasoftware engineeringtopicIdentifyingbestpracticewithrespecttospecificprocedures, technologies,methodsortoolsbyaggregatinginformation fromcomparativestudies ResearchquestionGeneric–relatedtoresearchtrends.Oftheform:which researchers,howmuchactivity,whattypeofstudies,etc.Specific–relatedtooutcomesofempiricalstudies.Oftheform: Istechnology/methodAbetterornotthanB? SearchprocessDefinedbytopicareaDefinedbyresearchquestionwhichidentifiesthespecific technologiesbeinginvestigated ScopeBroad–allpapersrelatedtoatopicareaareincludedbutonly classificationdataaboutthesearecollectedFocused–onlyempiricalpapersrelatedtoaspecificresearch questionareincludedanddetailedinformationaboutindivid- ualresearchoutcomesisextractedfromeachpaper Searchstrategy requirementsOftenlessstringentifonlyresearchtrendsareofinterest, forexampleauthorsmaysearchonlyatargetedsetofpub- lications,restrictthemselvestojournalpapers,orrestrict themselvestooneortwodigitallibraries
Extremelystringent–allrelevantstudiesshouldbefound. Usuallysystematicliteraturereviewteamsneedtousetech- niquesotherthansimplysearchingdatasources,suchas lookingatthereferencesinidentifiedprimarystudiesand/or approachingresearchersinthefieldtofindoutwhetherthey areundertakingnewresearchinthearea QualityevaluationNotessential.Alsocomplicatedbytheinclusivenatureof thesearchwhichcanincludetheoreticalstudiesaswellas empiricalstudiesofalltypesmakingthequalityevaluationof primarystudiescomplicated Importanttoensurethatresultsarebasedonbestquality evidence ResultsAsetofpapersrelatedtoatopicareacategorizedinavariety ofdimensionsandcountsofthenumberofpapersinvarious categories
Theoutcomesoftheprimarystudiesareaggregatedtoanswer thespecificresearchquestion(s),possiblywithqualifiers(e.g. resultsapplytonovicesonly)
54 4 Systematic Literature Reviews
analysis. They also conclude that raw data must be made available for meta-analysts, as well as additional non-published information from the primary study authors.
In a more recent literature review on the effectiveness of pair programming, Hannay et al. [73] conducted meta-analysis on data from 18 primary studies. They report separate analyses for three outcome constructs: quality, duration, and effort.
They also visualize the outcomes using forest plots.
4.6 Exercises
4.1. What is the difference between a systematic literature review, and a more general literature review?
4.2. What search strategies exist for primary studies?
4.3. Why should two researchers conduct some of the same steps in a systematic literature review?
4.4. What requirements are set on the primary studies to be included in a meta-analysis?
4.5. Which are the key differences between as systematic literature study and a
mapping study?
Case Studies
The term “case study” appears every now and then in the title or in the abstracts of software engineering research papers. However, the presented studies range from very ambitious and well-organized studies in the field, to small toy examples that claim to be case studies. The latter should preferably be termed examples or illustrations. Additionally, there are different taxonomies used to classify research.
The term case study is used in parallel with terms like field study and observational study, each focusing on a particular aspect of the research methodology. For example, Lethbridge et al. use field studies as the most general term [111], while Easterbrook et al. call case studies one of five “classes of research methods” [50].
Zelkowitz and Wallace propose a terminology that is somewhat different from what is used in other fields, and categorize project monitoring, case study and field study as observational methods [181]. This plethora of terms causes confusion and problems when trying to aggregate multiple empirical studies.
The case study methodology is well suited for many kinds of software engi-neering research, as the objects of study are contemporary phenomena, which are hard to study in isolation. Case studies do not generate the same results on, for example, causal relationships as controlled experiments do, but they provide deeper understanding of the phenomena under study in its real context. As they are different from analytical and controlled empirical studies, case studies have been criticized for being of less value, impossible to generalize from, being biased by researchers etc. The critique may be addressed by applying proper research methodology practices and accepting that knowledge is not only statistical significance [59,109].
The objective of this chapter is to provide some guidance for the researcher conducting case studies. This chapter is based on Runeson and H ¨ost [145] and more details on case studies in software engineering may be obtained from Runeson et al. [146]. Specifically, checklists for researchers are derived through a systematic analysis of existing checklists [79,145], and later evaluated by Ph.D. students as well as by members of the International Software Engineering Research Network and updated accordingly.
The chapter does not provide absolute statements for what is considered a ‘good’
case study in software engineering. Rather it focuses on a set of issues that all C. Wohlin et al., Experimentation in Software Engineering,
DOI 10.1007/978-3-642-29044-2 5, © Springer-Verlag Berlin Heidelberg 2012
55
56 5 Case Studies
contribute to the quality of the research. The minimum requirement for each issue must be judged in its context, and will most probably evolve over time.
The chapter is outlined as follows. We first introduce the context of case study research, discuss the motivations for software engineering case studies and define a case study research process in Sect.5.1. Section5.2discusses the design of a case study and planning for data collection. Section5.3describes the process of data collection. In Sect.5.4issues on data analysis are treated, and reporting is discussed in Sect.5.5.