Recent times have witnessed the emergence of software architecture evaluation as a sig- nificant practice and research area. These developments have prompted practitioners and re- searchers to create techniques, tools, and methods, which facilitate architecture appraisal,Over- hage et al.[2007] and Breivold et al.[2012]. Further, system architecture affects related de- velopment processes and acts as a centre for development. Consequently, architects generally qualify as leaders of the majority of development groups. Indeed, architecture remains one of the leading professions within the technology realm,Schlosser[2017].
In addition, quality attributes are a prime consideration during software architecting, de- signing, implementation, and testing for any engineering product that includes significant soft- ware components. No attribute is totally independent; each quality attribute supports, hinders, or has no effect on others. Existence of the ‘no effect’ option has low probability within the soft- ware development domain. This is because of the nature of software components’ interactive processes. Also, the interwoven relationship between quality attributes makes the trade-offs pro- cess necessary to some extent. More so, it increases the importance and complexity of capturing required qualities on an architectural level. Therefore, if one misses important requirements in the architecture phase, one is also almost sure of missing the same requirements through the next phases of designing and implementation. As a result, desired quality in the end-product will be unrealised. It is therefore important to get the architecture and its description correct for a successful result.
On the other hand, self-adapting systems, such as Artificial Intelligence (AI), have their own specific/particular sets of requirements, such as the ability to adjust to operational context changes,Cheng et al.[2009]. With AI systems, in particular, symbolic computing has come to signify a host of concerns regarding their architecture. One point of interest is to integrate the use of non-numeric symbols into a computer’s main functions. In essence, the goal is to have machines visualise environments, reason about them, based on their knowledge, and develop the best plan to be executed. This type of system increases the complexity of the architecture and its evaluation,Meystel et al.[2002]. Thus, the reasoning of quality attributes as they relate to software architecture is an important aspect. Although, it should be noted that AI researchers are too focused on specific concerns, which do not, in particular, involve software architecture as it generally applies. There are, however, a few exceptions to this generalisation, one of which is the Albus reference model 4-D/RCS. The architectural framework of this reference model tackles a range of behaviours, awareness, and knowledge attributed to humans. This open/op- erational control intelligent system architecture has been utilised in a variety of applications: autonomous ground vehicles, cleaning and debarring workstations, control systems for postal service stations, and more, Albus et al. [1996]. A second exception, is Roy Fielding (AI re- searcher), who developed the Representational State Transfer (REST) architectural framework for the purpose of understanding and evaluating architecture through the architectural styles used
2.2. INITIAL RESEARCH
in network-based application software, to satisfy the requirements of internet-scale distributed hyper-media systems,Fielding[2000]. Both architectures point to a strong relationship existing between SA and SPs.
In addition, every system needs to evaluate its software architecture to work properly. SAE methods are used to perform such assessments, which may also apply to Artificial Intelligence System (AIS).
However, it is general knowledge that software architecture is difficult to evaluate and com- pare objectively,Fielding[2000]. Thus, it is beneficial that requirements for quality attributes are identified in the early development phases of software, and are encapsulated within its ar- chitecture. The following reasons explain the importance of software architecture evaluation:
• The description of the software architecture is one of thefirst artefactsthat can be evaluated and analysed.
• The problems that are discovered during SA development processes can be fixed at con- siderably lower cost than if they are discovered in the testing and/or deployment stages, Clements et al.[2002a]. In the former, all that is involved would be modifying notations in the architecture. However, in the latter, source code may need to be changed on a massive scale, inducing needless cost and delay. The utilisation of full automation and modelling techniques may help reduce these costs, if they are available.
• Architecture evaluation at an early stage helps open the communication pathsbetween stakeholders and to develop a satisfactory system architecture which increases the success of projects.
• Architecture is at the centre of the development process. It includes decisions for the team structure, work division, configuration repository, documentation organisation, manage- ment strategies and development scheduling. An unsuitable architecture will cause a sig- nificant amount of disorder when it must be modified to address new concerns or defects that are uncovered in the early phase,Qin et al.[2008].
• The evaluation of software patterns (as a component of SA) aids developers in using and integrating them into other architectures, if need be, by utilising prior information on qual- ity attributes.
A concern that emerges here, is the quality of the resultant architecture when patterns with differing quality attributes are combined. However, the relationships between software patterns and quality attributes have been analysed and identified in a scientific manner, Freitas[2009],Kim et al.[2006] who used Alloy-Analyser as a tool, andZayaraz[2010]. Quality attributes, however, cannot be obtained in isolation, particularly within the con- text of complex systems. Whenever one attribute is achieved, another attribute is consequently affected. One such instance of this, is the relationship between performance and security. Typ- ically, the more security requirements that are applied to a system the more time is needed to process security checks, which in turn means a decrease in system performance, Bass et al.
[2013].
The current research derives from the significance of software architecture and its assess- ment, the worth of automation, the procedure for assessments, consistency of information and standardisation of notation, plus limitations of the current evaluation methods and techniques. Therefore, the outcomes are highly likely to add to the current body of software engineering knowledge.
In order to understand and improve Software Architecture Evaluation, there are some critical aspects that need to be addressed as reflected in the following questions: • How is SA being described? What are the limitations of the current SA description
methods?
• Do the current modelling methods have an effect on SA? What are the drawbacks and limitations of the current SA modelling techniques?
• How is SA being evaluated? What are the limitations of the current SAE methods? The rest of this chapter addresses these questions.
According toKlassen et al.[1998], aSystematic Reviewis “a review in which there is a com- prehensive search for relevant studies on a specific topic, and those identified are then appraised and synthesized according to a predetermined and explicit method”. The five common main steps in a systematic review are shown in Figure 2.1.
Figure 2.1: The main common steps for aSystematicreview.
However, The Integrated DEFinition language 0 (IDEF0)1diagram (Figure 2.2) and Ta- ble 2.1 draw the theme of the processes and strategies that been applied during this thesis re- search.
1
The IDEF0 (in brief), is a functional modelling approach designed for manufacturing and logistics activities, in order to describe processes, functions, development, business, and analysis within the engineering field,Buede et al.
2.2.
INITIAL
RESEARCH
Table 2.1: This is the review strategy that has been applied in Chapter 2, followed by Chapters 3, 4 and 5).
Steps of ‘evidence synthesis’
Comprehensive search and systematic review
roadmap Implantation and comments
1 Preliminary re-
search
In order to enhance SAE methodologies, the cur- rent SAE methods, gaps, and limitations must be identified. Thus, to scope the objective and to form specific questions, initial research should be performed.
Preliminary research has been performed, reported on, and discussed with, the su- pervisory panel and approved by them, through a Study Plan Report, on 30th January 2011 (12 pages) and (Thesis Proposal Review (TPR)-4th version,(44 pages). Some parts of the preliminary research were included in Chapter 2.
2 Objective
Is to combine strengths of critical comprehensive search processes and systematic review. It is to describe the state-of the-art to answer questions regarding SA and SAE. It can address broad or narrow questions to produce ‘best evidence syn- thesis’ and can provide answers to those ques- tions with critical descriptive analysis, either quantitatively or qualitatively.
A structured introduction and initial research are presented respectively in Sections 2.1 and 2.2, which will allow readers to assess quickly the relevance, quality, and generality of the review presented in Chapter 2.
3 Question formula-
tion
The rationale for the survey is based on spe- cific questions. Questions can be open-framed or closed-framed.
• The research questions were formulated and justified based on initial research in Section 2.2.
• SA and SAE are broad topics. However, several narrow subjects within (SA and SAE) were discussed, such as Formality, MDA, and Performance models, etc.
• Three important open-framed questions were listed, each one has been an- swered through major sections, which are: 2.3 SA; 2.4 Modelling SA; 2.5 SAE.
CHAPTER 2. BACKGROUND 4 Developing the research strategies and protocol
identify the method and strategy on how the re- search will be conducted. Prior planning helps to increase consistency, integrity, and visibility. Four important phases need to be considered dur- ing the review strategies are as follows:
1. Critical: assess selected information and articles.
2. Analyse: extract-opposing evidence from selected materials.
3. Synthesise, compare, and reveal the re- lationships between selected studies, con- cepts, or theories: differing characterisa- tions and descriptions, etc.
4. Evaluate and appraise practical use of the selected approaches.
I considered my Master research Almari [2010] as a pilot study for this review due to its relevance. The key points of the review strategy and protocol in this thesis are summarised in this table as follows:
1. The review title presented in Section 2.2. The reviewers include: my su- pervisory panel, and (external editor and proof-reader) as indicated in the ac- knowledgement section. The research duration is visible within the study plan report.
2. The objectives rationales pertaining to the questions and questions formula- tion are illustrated in Sections 2.1 and 2.2
3. Criteria for studies inclusion, search strategy, critical appraisal and data ex- traction, synthesise data, and reporting are all summarised in this table. For versioning, bibliography management, and document retrieval, I have used sev- eral applications during this research such as Curio, Papers, Concept-Draw Office, Bibdisk, Texmaker, Office, Excel, Word).
2.2.
INITIAL
RESEARCH
5 Conduct search and screening articles
The goal here is to find as many primary and sec- ondary studies that are related to the thesis ques- tion as possible.
No limitation on research evidence, which can be included (e.g., primary and secondary research).
The data selection concept in this thesis review follows a multistage iterative pro-
cess.The strategy used to quest relevant studies and sources include: journals (cover
company journals), conference proceedings, scientific databases, grey literature (i.e. technical reports), books, the Internet, etc.
Sources of evidence have been searched using the following methods:
• Trial searchers using (search-string) various combinations of terms and key- words derived from the research questions using boolean ANDs and Ors. • Consultations with my supervisor.
• Relevant studies references, strings, abstracts; and keywords. • Relevant reliable websites.
• Relevant Conferences materials.
Also, randomized controlled trials that investigated the research questions arena were used.
”Current software engineering search engines are not designed to support systematic literature reviews. Unlike medical researchers, software engineering researchers need to perform resource-dependent searches,Klassen et al.[1998]. However, the case now is much more mature than the situation in 1998.
CHAPTER
2.
BACKGROUND
6 Sift the search re- sult
The study selection criteria are proposed to clas- sify those studies, which provide direct evidence and strong relations to the research questions.
(SA, SA modelling, and SAE). Applications such as Curio, Papers, Concept-Draw, Bibdisk, Texmaker were utilised during the sifting process in order to manage the in- clusion process and to avoid duplications and confusion. The articles selection was based on the articles’ abstracts, key wording, and full text context that are relevant to the thesis questions.
Inclusion Criteria:
1. Any study that is relevant to the three main review questions regarding SA, SA modelling, and SAE.
2. Any study that addressed any of the research questions.
3. Any study addressed (SA and SAE) standardisation and automation processes in particular.
4. Empirical studies, experiments, surveys, models, standards, and frameworks that aimed to enhance and/or appraise (SA and/or SAE) methods.
5. Any information related to software patterns/styles documentation and eval- uation.
6. Related information published in English language. The following types of materials are excluded:
1. Unsourced information was excluded and removed.
2. Materials recommended as inappropriate by supervisory panel was excluded and removed.
3. Standards and frameworks that have been published after 2010 were excluded and will be recommended to be included in the future research.
4. Studies published in more than one journal/conference, the recent versions were used. Other versions have been excluded.
5. Studies published with the same aims, designs, methods, and results were excluded. Only the most common, reliable, popular, and complete studies were selected and included.
2.2.
INITIAL
RESEARCH
7 Critical appraisal & data extraction
Critical appraisal is the systematic evaluation for the studies included in the review and can be done using checklists. The data extraction process is designed to collect all the data required to address the research questions.
Data extraction:All the information needed that is satisfying inclusion criteria and relevant to the review questions was extracted. The data extracted include common information such as date of data, title, authors, publication details, additional notes available, etc.).
Critical appraisal processes were carried out based upon the information relevant to the research questions, which include the context facet + articles facet, and contri- bution facet. Also, included are studies’ design, methods used, and tools were. A Critical appraisal checklist with main five questions was created and employed. Each main question of the checklists comprises sub-questions:
1. Does the review address clear research questions?
2. Do the reviews’ results help to achieve the thesis main objectives? 3. What are the results of the review?
4. Are the review results providing a valid answer to the research questions? 5. Will the research results benefit the engineering field?
Supervisors reviewed all the research information, processes, and methods. They pointed out to any incorrect actions or issues within any aspects of the research. Regular meeting were scheduled and executed during the research.
Re-evaluating the selected studies after screening was conducted on iteration, in order to check their consistency to inclusion criteria.
8
Synthesis narra-
tively (and/or)
statistically
Trends in the literature, knowledge gaps, and clusters identified. Qualitative or quantitative synthesis of study results, where possible, using appropriate methodology or analysis.
Describing the findings in critical fashion with a positive argument towards the re- search area and question was conducted. The important aspects, issues, gaps, com- parisons, method mapping, and limitations around the three main subjects which are SA, SA-modelling, and SAE, were logically and critically discussed and iden- tified, as illustrated in Chapter 2. Visualising the findings through figures, tables, and mapping mechanisms were presented.
CHAPTER
2.
BACKGROUND
9 Report
The report should describe and categorise avail- able data relating to the research topic and ques- tions, identifying knowledge, advantages, disad- vantages, and gaps; identifying implications on the thesis research schema.
tique analysis) were used, in order to answer the review questions. Implications for this thesis were illustrated and summarised in Sections 2.6 and 2.7. The knowledge gaps and limitations were presented in Chapter 2, and recommendations for future research are made and reported in Chapter 7. Reporting the review in Chapter 2 was designed and catalogued carefully in order to point out to the current research problems and to be able to answer the research questions, while keeping the aim and structure of the whole thesis concrete, which is justified in Chapter 1 and Sec- tions 2.1 and 2.2. This thesis contribution to the identified problem is presented in Part II.