Spiral 2 Review and Plan

Context is included in the selection on a number of levels. Firstly it is included through many of the non-functional attributes in the ideal specification. Context is also represented in the tests, not only in the context schemas but also in the requirement for testing in the local environment - preferably the target context. A third level of localisation is the adaptability of the process to local organisational needs and tools.

Reuse

The process was influenced by existing processes, but did not reuse any process as a whole. However, there is some (internal) reuse through the specification template being utilised for a range of tasks throughout the Process.

5.7 Spiral 2 Review and Plan

An evaluation of this Spiral’s outcomes gave positive results against all of the stakeholder Win conditions. The activity diagram and areas for automation were identified and met project requirements for usability, context awareness and simplicity. In particular, the process is repeatable and self-documenting through the XML linkages between steps.

The GQM evaluation was positive on all questions, with only two areas rated as partially satisfied. However, Q2B1 (usability) would need the Process to be trialled in an organi-sational setting to be answered with confidence. This is beyond the scope of this project and may be considered in future work. Q2F1 (reuse) was also not completely satisfied, due to the creation of yet another selection process, where it had been hoped that an existing one could be used. However, the need for a process for this project indicates a gap in the literature which the CdCE Process addresses.

The work from Spiral 2 has undergone external review through conference papers and in an internal Faculty presentation. The contribution of this Spiral is the Process itself, particularly in the repeatability and self-documentation supported. The suitability for automation sets it apart from most selection processes, which are not ready for intelligent support (Ruhe, 2002).

Moving forward to planning for Spiral 3, the researcher identified parts of the Process to target for automation based on the manual use of the Process in Spiral 2. This resulted in a commitment to investigate test generation and filtering/shortlisting in subsequent

CHAPTER 5. THE CDCE PROCESS

Spirals.

5.8 Post-Spiral Update

Figure 5.6: Updates to the CdCE Process in later Spirals

The preceding Sections detail and review the work carried out in Spiral 2. Later Spirals have incrementally instantiated, extended and altered the CdCE Process as shown in Figure 5.6. This took place during the exploration of RE3 in Spirals 3 to 6. Impact on the Process was greatest in Spiral 3 (shortlisting approach), Spiral 5 (implementation of testing and evaluation) and Spiral 6 (support for shortlisting through ClassifierSuite) The final version of the CdCE Process will now be described, and is applied to a case study in Spiral 7 (Chapter 10). The final version of the inputs and outputs of all of the Steps are given in Table 5.10.

As the updates have impacted on every Step of the Process, they are now described on a Step by Step level.

5.8. POST-SPIRAL UPDATE

Step Inputs Outputs

Step 1 Desired attribute values Ideal specification (XML) Priority of attributes

Relationships between attributes

Formal specification of behaviour in Z notation

Step 2 Ideal specification (XML) Selected candidates (XML) Component repository information

(XML)

Analysis of component data

Update to ideal component specification (if applicable)

Step 3 Ideal specification (XML: Zspec) Abstract test cases (XML) Step 4 Abstract test cases (XML) Adaptation models

Candidate interfaces Adapted tests (XML) Step 5 Adapted tests (XML) Test results (XML)

Candidate executables

Step 6 Test results (XML) Evaluation metric scores (XML)

Step 7 Evaluation metric scores (XML) Ranked/ordered list of components (XML)

Ideal metrics (XML)

Step 8 Ranked list of components Report XML documents generated during the

se-lection process

Table 5.10: CdCE Inputs and Outputs

Step 1

There has been no conceptual change to Step 1, however the detail of the XML Schema describing the components and ideal component has changed. This includes the imple-mentation of improved data representation and ontologies and is discussed in Section 4.7.

The Schema was extended to include evaluation metrics developed in Spiral 5, which are described in Section 8.2.

Step 2

The shortlisting or filtering for a shortlist was targeted for automation. In Spiral 3, initial work considered using the Weighted Sum Method (WSM) or the Analytical Hierarchy Process (AHP) which were reported as widely used (Ncube and Dean, 2002). Limita-tions of these techniques led to investigation of alternatives, including Artificial Neural Networks and classifiers. This exploration is described in Chapter 6 and resulted in the adoption of machine learning classifiers for the shortlisting task.

The Process has strong support for iteration. In Spiral 6, an approach was taken to flatten the iteration through the selection criteria by creating a suite of classifiers

CHAPTER 5. THE CDCE PROCESS

to provided the results of using all combinations of criteria from the ideal specification.

While the ClassifierSuite does not remove the need to iterate back to Step 1, it does capitalise on the automation of the filtering and assists the developer in choosing among the given selection criteria when a trade-off is required. Full details of the ClassifierSuite are given in Chapter 9.

Step 3

Spiral 2 included the decision to generate abstract test cases from the specification.

The test generation itself was implemented in Spiral 5, and did not diverge from the expectations of Spiral 2. Some redefinition of the context-based testing (from Z schemas) was carried out to match the evaluation metrics developed in Spiral 5 (Section 8.2).

Step 4

The description of adaptation in Spiral 2 was quite high level. The expectation for an automated process for adaptation was not possible within time and resources. This resulted in a manual adaptation process being adopted in Spiral 5 (Section 8.4).

Step 5

The execution of tests was envisioned as utilising automation. As each testing environ-ment is different, it was decided in Spiral 5 to automatically generate tests and to leave the execution of them to the user. The tests themselves are recorded in XML to simplify their use in a test harness or other automated environment.

Step 6

Evaluation of tests is a step turning raw results into a set of compiled results. In Spiral 5 the specifics of the metrics were determined, guiding how the raw results populate indicators of performance in base and contextual tests. These are detailed in Section 8.2.

Step 7

As with shortlisting in Step 2, the ranking of components was considered for new strate-gies and approaches. During Spiral 2, AHP and WSM were under consideration. Having developed the classifier approach for shortlisting in Spiral 3, the same approach could

5.8. POST-SPIRAL UPDATE

Figure 5.7: Iterations possible within the CdCE Process

be applied to ranking based on the evaluation metrics. These metrics are extensible and interchangeable, however those used in this project align to the testing approach and the compiling of results from Step 6.

Step 8

The reporting of results was always considered to be a compilation of the documents from all steps of the Process. In the final version of the Process, this included the inputs and outputs from all Steps (XML presented through XSLT), along with descriptions of internal decisions, such as the choice of criteria and the shortlist using the ClassifierSuite in Step 2.

CHAPTER 5. THE CDCE PROCESS

Iteration

The initial version of the CdCE Process has one point of iteration in the activity diagram (Figure 5.2). The CdCE Process includes four iteration zones for the tuning of param-eters. The first is between specification and shortlisting (Steps 1-2), the second from specification to testing (Steps 1-6) and the third in ranking (Step 7). A fourth iteration is to repeat the entire process with modified criteria. Iteration is included to improve results and, as each change is documented, does not break the goals of structure and repeatability. The refinement of the specification for the shortlisting is expected to oc-cur on most selection tasks. All of these iteration options can be seen on the expanded Process diagram (Figure 5.7).

5.8.1 A Pattern for Component Selection

Over the course of the investigation, the CdCE Process was made flexible to allow the exploration of specific techniques at various phases of the selection. This generic approach makes it possible to consider the high level Process as a pattern for component (and more general) software selection. In instantiating the pattern, users may adopt the CdCE techniques, create their own techniques or use a blend of the two.

This description of the CdCE Pattern for Component Selection (see Table 5.11) uses the template provided by Gnatz et al (2002). Following the related theory for living software development processes, this description is at the meta-model level (Gnatz et al, 2002), with the CdCE Process itself at the ‘model’ level.

5.8. POST-SPIRAL UPDATE

Name: Context-driven Component Evaluation

Also Known As: N/A

Author: Valerie Maxville

Intent: As reuse of third party software increases, developers face the task of selecting between alternatives. This process provides a systematic, intuitive approach to the selection task.

Problem: The scenario is that the project requires one or more third party software items.

These may be standalone applications or software components to form part of the final system.

Context: The selection should begin in the requirements or design phases. For component-intensive systems, the pattern can be used in tandem with the rest of the system development. Developers will need to have information on the re-quirements (ideal component specification) and access to a software repository.

Solution: Eight Step Process:

Step 1 : Specify ideal component: The desired component is described using XML and Z notation. Any priorities between attributes should be included as well as relationships between attributes.

Step 2 : Shortlist candidates: The shortlisting step takes the ideal component specification and compares it to component information on repository websites.

Step 3 : Generate test cases: The behavioural specification in Z notation is used to generate tests to be applied to all the components. These are based on the interfaces and behaviour that are required.

Step 4 : Adapt tests: Given the shortlist of candidate components from Step 2 and the tests generated in Step 3, the tests are adapted ready to be run against each of the candidates.

Step 5 : Execute tests: Using an appropriate testing environment, all of the tests are executed for each candidate, and their performance is recorded.

Step 6 : Evaluate tests: This step takes the execution results from Step 5 and provides an evaluation against a set of metrics, ready for inclusion in the ranking classification.

Step 7 : Rank components: Information from the preceding steps is combined to determine a ranking or comparison of the components.

Step 8 : Report results: A report is generated to provide reasons for the deci-sions made and give information to assist with adapting the component to the target application.

Consequences: The pattern provides an intuitive approach to selection which was developed to be open to automation and tool support.

Known Uses: The pattern is instantiated as the CdCE Process. The Process includes a spec-ification template, tool support and process guidelines and has been applied in case studies. The Process is also able to be applied manually.

See Also:

Table 5.11: Pattern Definition: Context-driven Component Evaluation