Review protocol

The Figure 7 describes the review procedure which is applied for this systematic review.

Firstly, a literature review was performed to have certain background knowledge about methods/techniques which reduce QA effort. These approaches found during this review gave us an orientation and overview for doing a systematic review in combined QA approaches.

Before systematic review was conducted, a preliminary search is performed to explore whether there is already systematic review about combined QA approaches and to which extent. This step ensured that this systematic review is necessary and not a duplicate work.

After that, the research questions were defined according to the result of literature review. The search strategy including search term, data source, study selection criteria and study quality assessment are indentified before performing study searching. The search strategy was continuously modified during the evaluation until it was satisfied.

The review applies search strategy to get the search result from chosen databases and resources. The study selection criteria defined in search strategy are considered to choose the relevant paper based on

title/ abstract/ full-text. Thus the selected papers resulted from the searching will be reviewed to get the insight of the topic. The important data from those chosen paper will be extracted and analyzed. Based on the extracted data, the classification of combined QA approaches is proposed and the research questions are answered.

a. Search strategies

In order to perform the search in this review, four databases were identified. The main two chosen databases are Inspec and Compendex. These databases are the comprehensive database that contains millions of publication, mostly in engineering and computer science domain. Another reason for choosing Compendex and Inspect is Engineering Village interface for using Compendex and Inspect provides advance search feature a user friendly interface.

b. Search string

The search string is formed based on the key word derived from the research question. There are four attributes in the search string. Search attributes are derived from the thesis title: “Identification and analysis of Combined Quality Assurance approaches”. Table 4 shows the steps that are applied to form the search string and rationale for each step. Table 5 presents the relevant terms for each search attribute.

Table 4: Search string attributes

Search attributes Steps Rationale

Attribute 1 Synonyms for “static QA techniques", different common type of "static QA techniques”

To minimize the effect of heterogeneous terminology

Attribute 2 Synonyms of “software” To constraint the search in software engineering

Attribute 3 Synonyms of “dynamic QA techniques”, different common type of “dynamic QA techniques”

To minimize the effect of heterogeneous terminology

Attribute 4 Relevant term of “combine” Increase the coverage of search string

Table 5: Search string formation

Attribute Steps Relevant terms

1 Static QA techniques inspection or review or “static analysis” or “static quality assurance”

2 Software Software

3 Dynamic QA

techniques

test* or “dynamic quality assurance” or “dynamic analysis”

4 Combine combin* or integrat* or synergy or “trade off” or

complement*

Search string

Combine static and

dynamic QA

techniques for software quality assurance

((((inspection or review or “static analysis” or “static quality assurance”)) AND (((test* or “dynamic quality assurance” or “dynamic analysis”) and software))) AND ((combin* or integrat* or synergy or “trade off”)))

c. Study selection criteria

The selection criteria judge the studies based on the relevance of the paper after reading study title and abstract. If some of the research questions are mentioned in the paper, it is included in the selection.

Another main criterion to include the study is studies describing or comparing quality assurance approaches that combine static and dynamic QA techniques and give the evidence/evaluation data for the reviewed approaches. The study should be published within 20 years until now and not the whole proceedings of any conference. Only studies written in English are considered.

The review team (master student and supervisor) performs 3 steps to select the relevant paper. Firstly, the title of all the paper resulted from search string is read to decide whether the particular paper is relevant.

In the case that the reviewer is unable to make the decision on the selection based on the title, he/she marks the paper as "discuss". Those papers which are marked as "discuss" are brought forward to a review team (student and supervisor) discussion in order to gain the agreement on inclusion or exclusion for those papers. After selecting a pool of paper based on title, the similar steps are conducted for reading abstract and then conclusion. Finally, the final of inclusion paper is collected.

Inclusion criteria: Studies which introduce the QA approaches that combine static and dynamic QA techniques and have one of the following conditions:

• Studies which give evidence data/evaluation and motivation to combine static and dynamic QA techniques.

• Studies which describe the approaches, how to use, the context and requirement to apply that approach.

Exclusion criteria:

• Studies which are related to testing of other engineering field, not software testing such as microelectronics, medical, education, image processing..

• Studies which are related to testing of different hardware system (embedded system, analog-to-digital system...), not software testing in general.

• Studies which are too old and be published more than 25 years before or studies which are the proceedings of the whole conference.

• Studies which mention static QA techniques only or dynamic QA techniques only.

• Studies which are published in irrelevant conference or journal, for instance related to hardware, microprocessor, electronics and etc.

• Studies which are related to combined approach but not for quality assurance purpose.

• Studies which perform empirical study and compare static techniques with dynamic techniques but do not give suggestion or conclusion for combining static and dynamic techniques.

Study selection procedure:

The search strategies are depicted in Figure 7 and they are described as follows:

i. After determining the resources and initial keywords, trial searches are conducted using various combinations of search terms derived from the research questions.

ii. In order to assess the quality of the search string, some papers are picked randomly from the obtained list of papers according to the search criteria. If the search result includes the papers that are satisfy the search criteria, then the search criteria and search string are qualified. Otherwise, the search criteria and the keywords need to be refined.

Some papers that were already known to be relevant are used to check the coverage of the search keywords. If almost all of the known primary studies are found in the search, the search string is good enough. Otherwise, refining the search criteria and search strings is necessary.