2.5 Evaluating Software Architecture
2.5.2 Comparisons between current common architecture evaluation approaches
2.5.2.1 Scenario-based evaluation methods
Architecture evaluation methods are used to assess SA, where the specified quality attribute should be collected during both requirement gathering and evaluation phases, which then is represented by an initial or a series of architecture description candidates. Since quality attributes have different characteristics, defining the measurement of these attributes is essential before evaluating. This ensures that the attribute can characterise the capability of software in meeting the requirements.
The scenario-based method is the most notable of all architecture evaluation methods. It uses scenarios or hypothesised sets of the system’s uses or modificationsDobrica et al.[2002a]. A scenario covers a wide range of possible behaviours that may be done to the final system. In the context of architecture evaluation, scenarios are used to represent a concrete quality attribute. Some of the most popular evaluation scenario methods are discussed with critique as follows.
2.5. EVALUATING SOFTWARE ARCHITECTURE
2.5.2.1.1 Software Architecture Analysis Method (SAAM) was created in 1993 and pub-
lished in 1994. It is the first well-documented and carefully designed analysis method for archi- tecture analysisKazman et al.[1994]. During the early 1990s, when software architecture was still not widely accepted, it was already a remarkable effort,Qin et al.[2008]. Improvements were made later by getting more detailed descriptions and the financial managements system’s study cases by (Kazman - 1996), which have been cited byBass et al.[1998]). SAAM is a sim- ple, easy to learn and carry-out method that requires minimal amount of training and preparation.
1. Develop Scenarios Iterate 2. Describe Architecture(s)
3. Classify/Prioritize Scenarios
4. Individually Evaluate Indi- rect Scenarios
5. Assess Scenario Interaction
6. create Overall Evaluation
Figure 2.18: Activities in SAAM evaluation method, afterClements et al.[2002a].
SAAM is intuitive and simple; intuitive because it uses scenarios instead of quality attribute description to measure a software’s quality; and simple because it only considers the relation- ship between a scenario and architecture structure. Some evaluation methods, such as ATAM, were introduced after SAAM due to its control over many common quality attributes. SAAM also prepares a platform for stakeholders where they can discuss their ideas and concerns about the system’s blueprint and to resolve the understanding of deviations and incorrect architectures and/or designs. The general phases of evaluation, achievements for each phase, and the rela- tionship between them comprise SAAM’s steps. The primary steps are shown in Figure 2.18.
SAAM however has one pitfall since it does not provide a clear quality metric for the archi- tectural characteristics or attributes that are being analysed. Also, SAAM does not involve the use of tools, because there are no tools to support theSAAM evaluation method until recently. There are a few tools such as the Morale tool with limited support for SAAM,Zayaraz[2010].
2.5.2.1.2 Architecture Tradeoff Analysis Method (ATAM) is considered as SAAM’s ad-
vanced version. Aside from SAAM’s capabilities, ATAM also helps to better understand the trade-off to multiple relatives and even inconsistencies of quality requirements or targets. While
most experts were trying to enhance SAAM, its investors took notice of the relationship between targets reflected by scenarios and their effects on system construction.
Presentation
Reporting Analysis
Testing
Figure 2.19: The four phases of ATAM.
ATAM is built upon three areas: architectural styles; quality attribute analysis, and, SAAM, Qin et al.[2008]. It could be regarded as a hybrid technique because it uses questioning, scenar- ios, and measuring techniques. Questioning is based on architectural styles, quality attributes, and pre-existing questions. Scenario, is due to its main use of general and specific scenar- ios. Measuring by using quantitative outputs from reliability, performance and security models, Clements et al.[2002a].
ATAM’s main component consists of four phases that comprises of nine steps, in total. The preparation before the first phase and follow-up after an evaluation finish, can be considered as another two phases, which are not included in ATAM main phases. ATAM phases are repre- sented in Figure 2.19:
Each phase contains different steps as follows:
• Phase 1:Presentation– exchanging information through presentations, (3-Steps). • Phase 2: Investigation and analysis – assessing key quality attribute requirements with
respect to architectural approaches, (2-Steps).
• Phase 3:Testing– checking the results to date against the needs of all relevant stakehold- ers, (3-Steps).
• Phase 4:Reporting– presenting the results of the ATAM, (1-Step).
More explication for ATAM steps as follow:
2.5. EVALUATING SOFTWARE ARCHITECTURE
1. Present the ATAM – Evaluation methods are described to the participants, expecta- tions are set and questions are answered by the evaluation leader.
2. Present the business drivers – Business goals motivating the development effort and what the primary architectural drivers will be, are described by the project spokesper- son (ideally the project manager or system customer).
3. Present the architecture – Architecture is described by the architect, focusing on how it addresses the business drivers. In this step, I think that the explanations and jus- tifications of the architecture choices by an architect is not enough and needs to be accompanied by an evaluation mechanism and results for the architecture candidates, to be able to support the chosen architecture. Its this reason, one of many that en- courage me to do this research.
• Investigation and analysis
4. Identify the architectural approaches – Architectural approaches are identified but not analysed by the architect.
5. Generating the quality attribute utility tree; eliciting and specifying the level of sce- narios. Annotating with stimuli, responses, and prioritising the quality attributes that comprise system utility. For this step, the employing of the tactics that were intro- duced byBass et al.[2013], could be a very helpful approach.
6. Analyse the architectural approaches – Architectural approaches that address high- priority scenarios from Step 5 are elicited and analysed. Identifying architectural risks, non-risks, sensitivity points, and trade-off points are done in within this step.
• Testing
7. Brainstorm and prioritise scenarios – From the entire group of stakeholders, a larger set of scenarios is elicited, which are prioritisedviaa voting process involving all the stakeholders.
8. Analyse the architectural approaches – This step reiterates the activities of Step 6 but uses the highly ranked scenarios from Step 7. These scenarios are the test cases to confirm the analysis performed. Additional architectural approaches, risks, non- risks, sensitivity points, and trade-off points may be uncovered and documented through this analysis.
• Reporting
9. Present the result – Findings based upon the information collected during the ATAM evaluation is presented to the assembled stakeholders by the ATAM team. Fig- ure 2.20, shows the steps of the ATAM based on SEI description, Clements et al. [2002a].
• Present the ATAM • Present the Business
Drivers
• Present the Architecture
• Indetify the Architec- tural Approaches • Generate the Quality At-
tribute Utility Tree • Analyze the Architec-
tural Approaches
• Brainstorm and Priori- tize
• Analyze the Architec- tural Approaches
• Analyze the Architec- tural Approaches 1 Presentation 2 Investigation and Analysis 3 Testing 4 Reporting
Figure 2.20: Steps of ATAM based onClements et al.[2002a] description.
Kazman et al.[1999]. Merging several original steps, and complementing additional ones, en- hanced ATAM method.
There are two major improvements of ATAM to be observed.
First is the concern on how to realise when it is suitable to discontinue the generation of sce- narios. To address this concern, a set of quality attribute-specific questions has been developed by SEI, by which one can find that some useful scenarios are still missing and try to supplement them. More details about the questions may be found in SEI’s website6.
Second, is the adoption of Attribute-Based Architectural Styles (ABAS). ABAS, is a type of analysis-assistant tool, which helps the stakeholders to identify quality attributes brought by architectural styles. Since ABAS is a combination of stimuli, responses, and architectural decisions based on attributes and an analysis models, thus it may be described as an architectural style/pattern attached by attribute values, to reflect quality information. However, in the case of performance as an example, relevant information is not enough. An analytic framework will be used to facilitate analysis from the information gathered,Qin et al.[2008].
ABAS7is composed of four parts:
1. Problem description – a description of the problem that the structure solves.
2. Stimuli/responses – a characterisation and description of the stimuli that ABAS is designed to respond to, as well as a description of the quality-attribute-specific measures of the response.
3. Architectural style – relevant components to the quality attribute such as the set of com- ponent and connector types, the topology, a description of the patterns of data and control interaction among the components, and any properties of the components or connectors.
6
www.sei.cmu.edu
2.5. EVALUATING SOFTWARE ARCHITECTURE
4. Analysis – a quality-attribute-specific model that provides a method for reasoning about the behaviour of component types that interact in the defined pattern.
Since ATAM was created, it has experienced a continuous evolution and improvement. Some initial materials may be found in the works ofKazman et al.[1998] andBass et al.[1998], while ATAM ’s further detailed study cases can be found inBass et al. [2013] andClements [2003] works. The latest status of ATAM can be found on SEI’s website including the tutorials and support material. Figure 2.21 shows the latest conceptual flow of ATAM.
Analysis Tradeoffs Sensitivity Points Non-Risks Risks Risk Themes Distilled Info Business Drivers Quality Attributes Scenarious Architectural Plan Architectural Approaches Architectural Decisions Impacts
Figure 2.21: A conceptual flow of ATAM,SEI[2010].
2.5.2.1.3 Active Reviews for Intermediate Design (ARID) is an evaluation method for
partial architectures. It is situated between the intersection of two approaches. The first is scenario-based design review techniques, such as ATAM or SAAM that were discussed earlier. The second is active design reviews or ADRs. ARID is best suited for evaluating a partial archi- tecture/design in its formative stages. Figure 2.22 illustrates the two phases of this evaluation method.
Active Design Reviews (ADR)
ADRs are an effective technique for ensuring quality detailed designs in software. This method actively engages reviewers using carefully structured tasks, which avoids asking Yes/No questions. Reviews may be undermined by such questions, which enabling the reviewer to answer the questions without much consideration. In contrast, a sequence of exercises in this method that based on test concrete and not feigned understanding are asked by ADR to reviewers to utilise design.
Evaluating detailed designs of logical units for a software is the primary use of ADRs. Examples are modules and components. Its questions tend to address two things as illustrated below:
! "#$%&'()*&+$*,$-'$.$,/* ! 0,$12,$*&+$*#$/'3%*4,'$5'%3* ! 0,$12,$*&+$*/$$#*/6$%2,'7/* ! 0,$12,$*&+$*82&$,'29/*
0+2/$:;*
<$+$2,/29**
! 0,$/$%&*=<">* ! 0,$/$%&*&+$*#$/'3%* ! ?,2'%/&7,8*2%#*1,'7,'&'@$*/6$%2,'7/* ! =119)*&+$*/6$%2,'7/* ! AB882,'@$*0+2/$:C*
<$-'$.*
Figure 2.22: Steps of ARID based onClements et al.[2002a] description. 1. The quality and completeness of the documentation.
2. If the design’s provide sufficiency, fitness, and suitability concerning the required service. There are many other SA evaluating methods, which are mostly scenarios-based, each one does have its own unique focus and process such as:
• Architecture Level Modifiability Analysis (ALMA). • SAAM founded on Complex Scenarios (SAAMCS). • SAAM by Integration in the domain (ESAAMI). • SAAM Evolution and Reusability (SAAMER).
• Architecture Level Prediction of Software Maintenance (ALPSM). An overview of the above methods as follow:
• ALMA method:
The outcome of combining existing architecture level modifiability analysis approaches with scenario-based SA analysis approaches, which focus solely on modifiability is the Archi- tecture Level Modifiability Analysis (ALMA) method. ALMA method comprises five stages as illustrated in Figure 2.23; bear in mind, performing these steps is done with iteration,Bengtsson et al.[2004].
Kazman et al.[2005] did analyze ATAM and ALMA methods in terms of fifteen criteria; and they suggested several ways to improve both methods. Discussing these criteria is outside the scope of this research. However, there are some drawbacks for the ALMA methods such as::
• limits the attributes under inspection to modifiability.
• provides slight tractability support form the goal throughout the analysis outcome. • in general the authors interpretations for the results do not follow any specific techniques. More study and experiments could improve the ALMA method.
2.5. EVALUATING SOFTWARE ARCHITECTURE
Figure 2.23: The ALMA method five steps.
• SAAMCS, ESAAMI, and SAAMER methods:
SAAMCS, ESAAMI, and SAAMER as extended methods of SAAM. Where SAAMCS is focused on flexibility,Lassing et al.[1999]. The goal of SAAMCS is to use complex scenarios for risk assessment. On the other hand, ESAAMI and SAAMER developed byMolter et al. [1999] andLung et al.[1997] respectively , are designed for reusability,Dogru[2010]. In one hand, ESAAMI is an integration of SAAM within a specific domain with a reuse-based develop- ment process, which reuses the knowledge defined by SA’s models. However, ESAAMI cannot evaluate SA considering more than one QA at the same time. On the other hand, SAAMER provides a framework that collects stakeholders, scenarios, artefacts, SA, and QAs information, in order to support system reusability and evolution,Kim et al.[2010],Alebrahim[2017].
• ALPSM method:
ALPSM method’s main goal is to predict the size of change during maintenance using series of scenarios. Its focus on maintenance effort, defines a maintenance profile, and does not check if the architecture fulfill any other QA,Qin et al.[2008]. ALPSM consists of six steps as illustrated in Figure 2.24,Bengtsson et al.[2004]. However, ALPSM method does not consider reusability of the existing knowledge base,Dobrica et al.[2002b],Alebrahim[2017].
Figure 2.24: The ALPSM method six steps.
However, it is evident that most of the mature architecture evaluation methods, such as ATAM and SAAM are using qualitative methods that are normally applied through scenarios. It makes the resulting evaluation heavily skewed on the basis of selected scenarios and the inter- pretations given to required quality attributes. The generation of these scenarios is solely based on the vision of the stakeholders,Clements et al.[2002a] andZayaraz[2010].