Simulation-based Assessment

The assessment technique discussed in the previous section is rather static in that no executing dynamic model is used. An alternative is simulation-based assessment in which a high-level implementation of the software architecture is used. The basic approach consists of an implementation of the components of the architecture and an implementation of the context of the system. The context, in which the software system is supposed to execute, should be simulated at a suitable abstraction level. This implementation can then be used for assessing the behavior of the architecture under various circumstances.

Once a simulated context and high-level implementation of the software architec- ture are available, one can use scenarios from the relevant profiles to assess the rel- evant quality attributes. Robustness, for example, can be evaluated by generating or simulating faulty input to the system or by inserting faults in the connections between architecture entities.

The process of simulation-based assessment consists of a number of steps:

•

Define and implement the context: The first step is to identify the interfaces of

the software architecture to its context and to decide how the behavior of the context at the interfaces should be simulated. It is very important to choose the right level of abstraction, which generally means to remove most of the details normally present at system interfaces. For instance, for an actuator in the con- text of the system requiring a duty cycle to be generated, the simulated actuator would accept, for instance, a flow or temperature setting.

Especially for embedded systems where time often plays a role, one has to decide whether time-related behavior should be implemented in the system. For instance, when increasing the effect of the heater in the dialysis system, the actual water temperature will increase slowly until a new equilibrium is met. The software architect has to decide whether such delays in effects between actuators and sensors should be simulated or not. This decision is depending on the quality attributes that the software architect intends to assess and the required accuracy of the assessment.

Finally, for architecture simulation as a whole, but especially for the simulation of the system context, one has to make an explicit balance between cost and benefit. One should only implement at the level of realism required to perform the assessments one is interested in.

•

Implement the architectural components: Once the system context has been

Assessing Software Architectures

structed. The description of the architecture design should at least define the interfaces and the connections of the components, so those parts can be taken directly from the design description. The behavior of the components in response to events or messages on their interface may not be specified as clearly, although there generally is a common understanding, and the software architect needs to interpret the common understanding and decide upon the level of detail associated with the implementation.

Again, the domain behavior that is implemented for each component as well as the additional functionality for collecting data is again dependent on the quality attributes that the software architect intends to assess. For some quality attributes, additional architecture description data needs to be associated with the components. For instance, in the case of performance assessment, estimated execution times may be associated with operations on the component interfaces. In that case, generally a simulated system clock is required in order to be able to calculate throughput and average response time figures.

•

Implement profile: Depending on the quality attribute(s) that the software

architect intends to assess using simulation, the associated profile will need to be implemented in the system. This generally does not require as much imple- mentation effort as the context and the architecture, but the software architect should be able to activate individual scenarios as well as run a complete profile using random selection based on the normalized weights of the scenarios. For example, in virtually all cases, the use profile needs to be implemented. The software architect generally is interested in performing individual use scenarios to observe system behavior, but also to simulate the system for an indefinite amount of time using a scenario activator randomly selecting scenarios from the profile.

•

Simulate system and initiate profile: At this point, the complete simulation,

including context, architecture and profile(s) is ready for use. Since the goal of the simulation is to assess the software architecture, the software architect will run the simulation and activate scenarios in a manual or automatic fashion and collect results. The type of result depends on the quality attribute being assessed.

It is important to note that for several quality attributes, the simulation will actu- ally run two profiles. For instance, for assessing safety, the system will run its use profile in an automated manner, and the hazard scenarios will, either manu- ally or automatically, be activated. Whenever a hazard scenario occurs, data concerning the system context is collected. For example, in the dialysis system example, all values relevant to the safety of the patient, e.g. blood temperature, concentrate density, air bubbles, heparin density, etc. are collected from the

Simulation-based Assessment

simulation to see if any of these values, perhaps temporarily, exceed safety boundaries.

•

Predict quality attribute: The final step is to analyze the collected data and to

predict the assessed quality attribute based on the data. Depending on the type of simulation and the assessed quality attribute, excessive amounts of data may be available that need to condensed. Generally, one prefers to automate this task by extending the simulation with functionality for generating condensed output or using other tools.

To give an example, for performance assessment, the system may have run tens (or hundreds) of thousands of use scenario instances and collected the times required to execute the scenario instances. All this data needs to be condensed to average execution times per scenario, perhaps including a standard deviation. This allows one to make statements about system throughput, scenario-based throughput and average response times of individual scenarios.

Simulation of the architecture design is not only useful for quality attribute assess- ment, but also for evaluating the functional aspects of the design. Building a simu- lation requires the engineer to define the behavior and interactions of the architecture entities very precise, which may uncover inconsistencies in the design earlier than when using traditional approaches. In our experience, it is extremely useful to be forced to express exactly what the functionality of an architecture com- ponent should be. Simulation has been used in all three examples systems described in chapter 2 and the experiences have been very positive. Although there is an over- head involved in the implementation of, especially, the system context, our experi- ence is that the advantages easily outweigh these.

The fact that an executable specification of the system is available early in the design process often proves to be highly relevant. It is important to note that it is possible to have a simulation of the system available during the whole design pro- cess. This is achieved through iterative refinement of the simulated context and the system implementation. Using this approach, the software architect iteratively details the design and the context in which the design operates until the context is no longer simulated, but the actual context. If this level is achieved, the system implementation is also complete. It is possible to use parts of the actual system con- text early in the design and use simulated parts where the system is not sufficiently detailed yet. Although our experience is that this approach is very interesting for, especially, the implementation of embedded systems with high availability and strict safety requirements, it is outside the scope of this book. We refer to, among others, [ref] for more elaborate discussions.

Assessing Software Architectures

Simulation complements the scenario-based approach in that simulation is particu- larly useful for evaluating operational quality attributes, such as performance of fault-tolerance by actually executing the architecture implementation, whereas sce- narios are more suited for evaluating development quality attributes, such as main- tainability and flexibility.

Simulation can be used for the assessment of reliability in two ways. First, using component reliability figures, one can simulate component failures during auto- mated execution of the usage profile and collect reliability figures for the architec- ture based on this data. Secondly, some research results, e.g. [ref-claes-SDL] has shown that correlations exist between the reliability of specifications and the sys- tems implemented based on those specifications. Thus, the reliability of the archi- tecture implementation in the simulation gives an indication of the reliability of the final system.

Although simulation is primarily suited for the assessment of operational quality attributes, the implementation of the architecture in the simulation can be used to evaluate development quality attributes, such as maintainability. Once the simulat- tion is available, the software architect can actually implement scenarios from the maintenance profile and measure the required effort and identify the affected com- ponents and the extent of change. Using this data, one can extrapolate the required effort for the complete system.

Finally, the accuracy of simulation-based assessment depends on a number of fac- tors. First, the accuracy of the profile used to assess the quality attribute. Second, it is dependent on how well the simulated system context reflects real world condi- tions. Finally, the relation between the architecture implementation and the imple- mentation of the final system. Consequently, the factors influencing accuracy are considerably more than for scenario-based assessment, but the early availability of an executable implementation of the system has, as discussed, several advantages as well.

*** example - measurement system paper? Main problem, nice functional model, but what quality attribute? ***

Summarizing, simulation-based assessment makes use of an executable model of the software architecture and a simulation of the context of the system. Using the model and its context, one can execute the profile for the quality attribute that is to be assessed. Using the data collected during profile execution, one can predict the level of quality attribute for the software architecture. The simulation-based assess- ment process consists of the following steps:

Mathematical Model-based Assessment

•

Define and implement the context

•

Implement the architectural components

•

Implement profile

•

Simulate system and initiate profile

•

Predict quality attribute

FIGURE 12.Simulation of a beer can inspection system

In document Design and Use of Industrial Software Architectures (Page 67-71)