We feel confident that the prototype based evaluation approach is useful for assessing the performance characteristics of an archi- tecture component and also for evaluating the performance of architecture models derived from proposed software architec- tures. New architecture models are easily implemented using the evaluation support framework and the amount of reuse, both in code and analysis tools makes the creation of a support frame- work and analysis tools worth the effort.
The support framework separates the component that we are evaluating from the architecture model, making it possible to compare alternative components in a consistent way as the data for the evaluation is gathered in the same way independently of the component.
A concern that can be raised against the use of an evaluation sup- port framework is that since a message has to go through the framework classes before it reaches the component that we are evaluating there is an added delay. In our case study we found that the delay between that the worker sent the message and that the message was actually sent by the interaction component was quite small. The framework added between 0,1 to 0,3 ms to the time it took to send a message.
An Approach for Performance Evaluation of Software Architectures using Prototyping
Future Work 87
4.8
Future Work
We plan to continue to use the test framework and the test appli- cations and try to evaluate other quality attributes such as scala- bility and maintainability.
The test-framework and applications will be used to perform fol- low-up evaluations at Danaher Motion Särö AB in order to see how the communication component develops, and ultimately to compare how well the original estimations match the perform- ance of the final system.
4.9
Conclusions
In this chapter we have described the prototype based architec- ture evaluation approach and the steps that it consists of. The approach is based on the simulation based evaluation approach but adds mainly the construction of an evaluation support frame- work and a clearer focus on iteration during the evaluation. The use of the evaluation support framework simplifies the imple- mentation of alternative architecture models, makes consistent gathering of data simpler, and makes it possible to evaluate alter- native implementations of an architecture component.
The evaluation approach has been used to evaluate the perform- ance characteristics of a communication component in an AGV system architecture. The evaluation resulted in that a perform- ance problem was identified and that two additional questions were evaluated as well (Portability and performance differences between Windows 2000 and Linux). Results from the case study were also used to validate the evaluation approach once the new system was stable, and showed that it produced accurate results. The case study illustrates the steps within the evaluation process and can be seen as a guide for performing a prototype based eval- uation.
Acknowledgments
This work was partly funded by The Knowledge Foundation in Sweden under a research grant for the project "Blekinge - Engi- neering Software Qualities (BESQ)" (http://www.ipd.bth.se/
88 Conclusions
besq). We would also like to thank Danaher Motion Särö AB for providing us with a case for our case study and many interesting discussions and ideas.
Introduction 89
C
H A P T E R
Evaluating Software Quality
Attributes of Communication
Components in an Automated
Guided Vehicle System
Frans Mårtensson, Håkan Grahn, and Michael Mattsson
5.1
Introduction
The size and complexity of software systems are constantly increasing. It has been identified that the quality properties of software systems, e.g., performance and maintenance, often are constrained by their software architecture [12]. The software architecture is a way to manage the complexity of a software sys- tem and describes the different parts of the software system, i.e., the components, their responsibilities, and how they interact with each other. The software architecture is created early in the development of a software system and has to be kept alive throughout the system life cycle. One part of the process of creat-
90 Introduction
ing a software architecture is the decision of possible use of exist- ing software components in the system.
The system we study in this chapter is an Automated Guided Vehicle system (AGV system) [24], which is a complex distrib- uted real-time system. AGV systems are used in industry mainly for supply and materials handling, e.g., moving raw materials, and finished products to and from production machines. Impor- tant aspects to handle in such systems are, e.g., the ability to auto- matically drive a vehicle along a predefined path, keeping track of the vehicles’ positions, routing and guiding the vehicles, and collision avoidance. The software in an AGV system has to be adaptable to quite different operating environments, e.g., iron works, pharmacy factories, and amusement parks. More impor- tantly, the system may under no circumstances inflict harm on a person or object. The safety and flexibility requirements together with other quality- and functional requirements of the system make it a complex software system to create and maintain. In the system in our case study, the software in the vehicle can be divided into three main parts that continuously interact in order to control the vehicle. These parts communicate both within proc- esses as well as between processes located on different comput- ers.
In this chapter we evaluate two communication components used in an existing AGV system and compare them to an alternative COTS (commercial-off-the-shelf) component for communication [79] that is considered for a new version of the AGV system. We evaluate three quality attributes for each of the components: per- formance, maintainability, and portability. We use three proto- types built using a prototype framework to measure the performance of each component. Both intra-process as well as inter-processes communication are evaluated. The communicat- ing processes reside both on the same computer and on two dif- ferent computers connected by a network. We measure the maintainability of the three components using the Maintainability Index metric [69]. We also discuss qualitative data for the porta- bility aspects of the components.
The evaluation is performed in an industrial context in coopera- tion with Danaher Motion Särö. The usage scenarios and archi- tecture description that are used during the evaluations have been developed in cooperation with them.
Evaluating Software Quality Attributes of Communication Components in an Automated Guided Vehicle System
Background 91
Our results indicate that the performance of the COTS compo- nent is approximately half the performance of the in-house devel- oped communication components. On the other hand, using a third party COTS component significantly reduce the mainte- nance effort as well as increase the functionality. Finally, all three components turned out to be portable from Windows XP to Linux with very little effort.
The rest of the chapter is organized as follows. Section 5.2 presents some background to software architecture, architecture evaluation, and automated guided vehicle systems. In Section 5.3 we introduce the components and the quality attributes that we evaluate. We present our evaluation results in Section 5.4. In Sec- tion 5.5 we discuss related work and, finally, in Section 5.6 we conclude our study.