• An inventory of the research area of performance modelling for system-level
design has been made. This inventory identified the requirements for obtain- ing credible performance results and revealed several deficiencies in state of art performance analysis techniques, modelling languages, property specification languages, as well as system-level design methods and tools. With the inven- tory, it has become possible to identify where viable combinations of existing techniques, formalisms, methods and tools need to be extended/modified to ensure obtaining credible performance results during system-level design.
• Evaluating long-run average performance metrics often requires to take a cer-
tain condition into account. A simple example of such a conditional long-run average is the average duration of processing packets by a telecommunica- tion system. For this performance metric, the value of the variable denoting the most recent processing time must only be taken into account when com- pleting the processing of a packet. Nevertheless, straightforwardly applying classical Markov-chain based performance analysis techniques in this case re- quires to consider the value of the variable in all states of the system. With the proposed reduction technique, it has become possible to immediately dis- regard reward values in irrelevant states in the case of analytical computation as well as simulation-based estimation of conditional long-run averages. Es- pecially in the latter case, performance evaluation speed is improved consider- ably, thereby greatly increasing the system complexity that can be handled.
• Many common long-run average performance metrics can be defined as an
algebraic combination of simple (conditional) long-run averages. Examples of such complex performance metrics are the average occupancy of a buffer and the throughput of an on-chip bus. Although classical Markov-chain based performance analysis techniques provide ample means for computing these performance metrics analytically, a general approach for deriving the accuracy of estimation results obtained for complex long-run averages is missing. With the proposed algebra of Confidence Intervals, the accuracy of estimation re- sults for complex (conditional) long-run averages can be analysed without the necessity to explicitly derive a Confidence Interval. Hence, the algebra of Con- fidence Intervals has enabled analysing the accuracy of estimation results for many common complex long-run averages for which this was not yet possible.
• Markov-chain based performance analysis techniques for estimating long-run
averages require to specify a state that determines the beginning of a regener- ative cycle of independent behaviour. Usually, it is impossible to specify such a recurrent state in advance and hence, it is desirable to identify a recurrent state during simulation. One could base this identification on detecting the re- visiting of a state. Unfortunately, this approach is often prohibitively complex for industrial hardware/software systems and hence, it is more practical to define a certain recurrence condition that enforces the beginning of a regenera- tive cycle. Two approaches for defining such recurrence conditions have been proposed. One approach is generally applicable and matches the commonly used batch-means technique. This technique has however some theoretical and practical deficiencies. The other approach relies on knowledge about the behaviour of a specific component of the system, thereby yielding more credi-
ble estimation results. With these approaches, it has become possible to apply the simulation-based estimation techniques proposed in this thesis in practice.
• Instead of expressing the behaviour of a system in the form of a Markov chain,
it is more convenient to use an expressive modelling language such as POOSL. To enable applying Markov-chain based performance analysis techniques in this case, POOSL models have been mathematically related to Markov chains in the context of a reflexive approach for specifying monitors. This framework for reflexive performance analysis provides a sound basis for implicit applica- tion of Markov-chain based performance analysis techniques while executing a POOSL model that is extended with performance monitors. Hence, the pro- posed framework has enabled a rigorous evaluation of performance metrics based on models constructed with an expressive modelling language.
• Four (conditional) long-run average performance metric types have been iden-
tified as most common for hardware/software systems. Based on the frame- work for reflexive performance analysis with POOSL, the proposed Markov- chain based estimation techniques and the approaches for approximating the beginning of regenerative cycles, library classes for analysing the accuracy of the four performance metric types have been developed. With these per- formance monitor classes, it has become possible to estimate long-run aver- ages without knowing all details of the mathematical theories involved. The credibility of estimation results obtained when using the performance monitor classes has been assessed with an experiment. The experiment revealed that for the estimated performance metrics (one of each type), utilising knowledge about the re-occurrence of local behaviour to define a recurrence condition is favorable over using the current implementation of the batch-means technique.
• The SHE method has been extended with a framework for constructing and
validating performance models with POOSL. This framework integrates sev- eral performance modelling guidelines with the existing techniques, guidelines and tools for SHE. The SHE method distinguishing the phases of formulation, formalisation and evaluation. The applicability of the SHE method has been improved by defining a UML profile that facilitates deriving POOSL models from stereotyped UML diagrams in the formalisation phase. In addition, sev- eral modelling patterns have been presented next to guidelines for extending POOSL models with monitors and for their validation. With the proposed ex- tensions to the SHE method, a system-level design method has become avail- able that allows rigorous evaluation of the performance (and correctness) of design alternatives for industrial hardware/software systems.
• A number of academic and industrial case studies has been performed. These
case studies were very valuable for identifying the theoretical and practical issues involved in the area of performance modelling and also for assessing the developed techniques and method. Next to successfully applying the pro- posed performance analysis techniques and system-level design method, the actual performance results for the different industrial case studies have been valuable for thoroughly understanding the working of the involved systems and for founding decisions between design alternatives for these systems.