Specification Exemplar Application

The specification exemplar supports the RE-tool evaluation in two ways. First, it provides the evaluation criteria. Second, the specification exemplar is used as the try- on instrument to compare different tools.

The application of the specification exemplar and the evaluation frameworks leads to the guidelines which summarise the RE-tool evaluation approach (R-TEA) in Figure 6.1. As discussed in section 4.3 the evaluation process involves two actor groups: evaluation team, which manages the process, and potential users, who perform the evaluation activities. R-TEA includes six general guidelines:

1. Preparation of a requirements specification. The first phase consists of analysing the specification exemplar and adapting it to the user needs:

a) The evaluation team performs elicitation. The purpose is to highlight the most representative environmental needs. The starting elicitation point is the specification exemplar which consists of both evaluation frameworks. Elicitation should also highlight the level of organisational maturity, the work processes and product modelling paradigm. During the elicitation activities, the user should also specify the additional RE-tool requirements which are not described in the frameworks.

b) The purpose of the prioritisation activity is to determine the most important features and activities for the RE-tool. The evaluation team could apply different prioritisation algorithms (Analytical Hierarchy Process (AHP) (Saaty, 1980), weighted scoring method (WSM) (Kontio, 1996), Planning game).

c) Based on elicitation and prioritisation results, the evaluation team prepares the requirements specification, which contains the most needed user needs and RE-tool requirements, and where they are organised according to their priorities. The requirements specification for the RE-tool could contain the following RE-tool requirements:

− the framework features which are selected without change from the frameworks and adapted as the RE-tool requirements.

− the framework features but these features are changed according to the user needs to the RE-tool requirements.

− new RE-tool requirements, not defined in the frameworks, but elicited from the user work practice.

Figure 6.1 RE-tools Evaluation Approach

2. Selection of business parties involves the investigation of the RE-tool market according to the external requirements. The evaluation team requests trial and demonstration RE-tool versions from the business parties.

Next, phases 3, 4, and 5 describe the evaluation of each of the RE-tool, selected in the phase 2. All three following phases corresponds to the assessment of the tool compatibility as shown in Figure 4.3.

3. Investigation of the functional requirements contributes with the functionality evaluation of the RE-tool candidate. In order to test the functional RE- tool requirements an evaluation technique(s) is used. The evaluation techniques provide guidelines for getting familiar with the RE-tools selected for evaluation in

Specification Exemplar 1. Preparation of a requirements specification 2. Selection of the business parties RE-tools candidates Requirements Specification 6. Decision about the RE-tool(s) 4. Investigation of the process requirements 5. Investigation of the product requirements Evaluation results

from phases 3, 4, and 5

Evaluation technique(s)

(e.g. specification exemplar, RE- tool tutorial, evaluation scenario)

RE-tool(s) found Investigation of RE-tools No Yes 3. Investigation of the functional requirements

RE-TOOL EVALUATION APPROACH

Table 6.1 Evaluation Techniques

Evaluation

technique Advantages Disadvantages

Requirements specification(or specification exemplar)

− Is prepared in phase 1 according to specification exemplar (or specification exemplar itself); − Is prepared according to user familiar domain; − Emphasises the same evaluation issues.

− No guidance during the RE-tool assessment.

RE-tool tutorial

− Emphasises on teaching the RE-tool functionality; − Is received together with RE-tools;

− Provides guidance during the RE-tool assessment.

− Tool evaluation could not be comparable if tutorials emphasise different tool functionality.

Evaluation scenario

− Emphasises on teaching the RE-tool functionality; − Emphasises the same functionality for all RE-

tools;

− Provides guidance during the RE-tool assessment; − Is prepared for all RE-tool candidates on the same

domain;

− Domain could be related to the user work practice.

− Preparation is time demanding; − Preparation is domain

knowledge demanding.

detail tool evaluation, but the evaluation itself takes more time than using one evaluation technique. Requirements specification as evaluation technique is used in the example in section 6.2.3. Other two evaluation techniques are applied in case studies described in Chapter 7.

After testing all the RE-tool candidates, users fill in the evaluation forms. The evaluation forms where tool features are mapped with the requirements specification, are prepared for each tool. Evaluation form is some kind of questionnaire where users set the evaluation marks in a predefined scale. Evaluation form should also allow writing comments on any related evaluation issue.

When the users are performing the RE-tool testing activities, the evaluation team should perform observation activities. Observation has two purposes. First, the evaluation team could react to the difficulties and problems which arise during tool testing. Evaluation team could provide guidance and teaching remarks, too. Second, the observation could provide the useful information about RE-tool non-functional product characteristics, like RE-tool usability, reliability, and performance.

4. Investigation of the process requirements. Non-functional process requirements are analysed in correspondence to the relationships (Figure 5.4). The phase results with the lists of inadequacies between the user RE processes and the RE- tool support for the RE process. The inadequacies are considered in the phase 5 as the maintainability requirements.

5. Investigation of the product requirements. Non-functional product

requirements are evaluated in the usability, performance and reliability tests. The observation, made in phase 3, also contributes with the evaluation of the non- functional product requirements. The tool maintainability requirements, selected in phase 4, should be investigated together with the business parties’ support. The

evaluation team should investigate which maintainability requirements could be fulfilled by the tool users, and which should be redirected to the RE-tool vendors.

6. Decision about the RE-tool selection is made after summarising the results from phases 3, 4 and 5. Phase 3 contributes with the evaluation of the functionality, phase 4 addresses how the tool fits the working process and what is the need for maintenance, and phase 5 provides the evaluation of the non-functional product features. After the result analysis, one of three possible decisions should be made: a) The users start using the “best-evaluated” RE-tool without changing their RE

process.

b) The users start using the “best-evaluated” RE-tool, but they have to reconsider their RE process.

c) The “best-evaluated” evaluated tool is not suitable for the users and they need to repeat the RE-tool evaluation (reconsider the requirements specification, and/or search for other RE-tools candidates).