Open Source Component Evaluation Approaches

Many tools have been developed just for evaluation of OSS. These tools can be suc- cessfully used as such, modified, or even some evaluation results of well-known software can be found. A comprehensive overview of OSS evaluation methods has

5.1. RELATING OSS EVALUATION TO SPL 51 been presented in an article by David A. Wheeler [134]. In addition to collecting ref- erences to existing evaluation methods, Wheeler proposes a method for comparing OSS components. The method consists of four steps: identify candidates, read existing

reviews, briefly compare the leading program’s attributes to your needs, and then perform an in-depth analysis of the top candidates [134]. This method is meant for OSS com-

parison. However, in the case of evaluation of OSS for reuse in software solutions discussed in this chapter there can often be one candidate that should be evaluated for its quality and for possible risks introduced to the solution. The functionality evaluation is naturally important as well. However, some components can be evalu- ated for their potential use in future software solutions without explicit requirements for functionality. In such cases the functionality evaluation can be limited to record- ing the provided functionality. Despite a different purpose, the evaluation method presented by Wheeler includes some criteria that can be applicable in any context. For instance, the method suggests criteria like support, security, or legal/license is- sues, among other criteria [134]. Those criteria are also used in the context of OSS component evaluation for commercial software solutions, as proposed in this thesis in Section 5.2.

Other prominent evaluation methods include, for example, Qualification and Selection of Open Source software (QSOS) [7, 114], Open Source Maturity Mod- els by Navica [90] and Capgemini [23], or Business Readiness Rating (BRR) [20]. QSOS [7, 114] consists of four steps:

1 definition, which specifies the scope and context of the evaluation,

2 evaluation, which covers functionality and risks for users and service providers, 3 qualification, weighting the criteria according to an axis specified in evaluation,

and finally

4 selection, which can be strict or loose depending on needs.

The QSOS method is based on a system of weights and point scores that can be ad- justed depending on the needs. Additionally, a QSOS method is under GNU license itself including evaluation reports that are available at their website1 and can be a valuable source of information for component evaluation.

Another distinctive evaluation method is Business Readiness Rating (BRR) [20], which aims at finding out the suitability of software for usage in a commercial con- text. BRR proposes a set of criteria that can be adjusted depending on requirements, as well as templates using different weights for various metrics and score points. Similarly to QSOS BRR offers ready templates for certain OSS on their website 2_,

which can be a helpful starting point for evaluation. Other methods, namely Open Source Maturity Models by Navica [90] and Capgemini [23], provide their own con- tribution to OSS evaluation. For instance, Capgemini’s Open Source Maturity Model (OSMM) [23] uses four criteria types and 12 criteria to assess maturity, compare, and select OSS. Interestingly, however, some researches found that formal evaluation methods are not widely used in industry, at least such an observation was reported

1_{http://www.qsos.org}

52 5. DEVELOPMENT - COMPONENT EVALUATION

in the Norwegian cases reported by Hauge et al. [57]. This report has not been dis- cussed in publication [P3]. However, it provides an additional and recent account of OSS evaluation in commercial software companies. Hauge et al. report that in the investigated cases OSS evaluation was rather informal, and often an important role was played by recommendations from a social network or reviews found on the Internet [57, p. 44].

Generally, all of the above-mentioned OSS evaluation methods provide a valu- able source of possible criteria to be used for evaluation. Moreover, they provide in many cases ready-made evaluation reports. Some criteria used in these evalua- tion frameworks were also used in the criteria for OSS component evaluation pro- posed in this thesis, as listed in Section 5.2. Additionally, the OSS component evalu- ation framework advises to use any existing evaluation results, such as ready-made evaluation reports, as a source of information. The main difference between the official evaluation frameworks and the proposed OSS component evaluation frame- work (see Section 5.2) is the fact that the latter uses a specific set of criteria that was adjusted to a particular context. Additionally, the above-mentioned OSS evaluation methods use various point systems for evaluating individual criteria, while the OSS evaluation framework proposed in this chapter does not use any points for criteria evaluation and relies on the experience and judgement of the evaluator.

At a general level the evaluation process of an OSS component can be abstracted as depicted in Figure 5.2. The process typically starts from requirements gathering, and then selection of relevant software is done. Next, functional evaluation and non-

functional evaluation are performed. Finally, a chosen component can be integrated

with the whole system. These steps are quite general. However, they set the stage for combining typical OSS evaluation and certain aspects of SPL, which are presented in the subsequent section.

Figure 5.2 A typical evaluation process [P3, p. 12]