We performed an empirical evaluation of the modelling methodology aimed at addressing its scala- bility with respect to the effort involved in the modelling process. The modelling process must be scalable in practice, and only a realistic empirical evaluation can help us to make this assessment. The empirical evaluation aims to respond to the following research question:
Chapter 3. Modelling Methodology
The size of the created model and constraints for an actual, representative data processing system is a surrogate measure for modelling effort, since actual effort is largely dependent on skills and experience and therefore not a generalisable measure. Of course, determining if the size of a model is acceptable is subjective. An assessment must therefore be made in light of typical system models and based on the experience with project engineers.
3.6.1
Subject selection
This empirical evaluation makes use of the SES-DAQ system described in Chapter 2.
3.6.2
Approach application and data collection
To perform the empirical evaluation, we first studied the SES-DAQ system using design and test documents and held several modelling sessions with the system testers and developers. The model and constraints for the system were created, in an iterative manner, following the modelling methodology defined in Sections 3.2 and 3.3.
For this evaluation, IBM Rational Software Architect (RSA) was used to create the UML class diagrams and the OCL constraints. Alternatively, there are free software packages available that could also be used (e.g. Papyrus UML).
3.6.3
Results
In this section, we discuss the results of the empirical evaluation to answer our research question. As mentioned before, the size of the model and constraints can be a surrogate measure to estimate the effort needed to follow our modelling methodology in a specific context. We report model size rather than modelling time because the time needed for modelling is largely dependent on the person’s domain knowledge and expertise in modelling and in OCL, and is therefore expected to vary from context to context.
We count and report the number of classes, attributes, associations (which include compositions) and generalisations of our case study system in Table 3.2. The results show that the size of the model, with 68 classes overall, seems acceptable given typical system model sizes and the size of the SUT (53.5 KLOC). Of course, as mentioned before, such an assessment is subjective. Therefore, we also rely on the feedback of the system’s developers and testers when they were presented with the final model and constraints; since the system is developed by third parties, the modelling methodology allows for a high-level view of input and output constraints. The feedback we received is that the size of the model and constraints are reasonable compared to the benefit of defining the constraints used for test validation and oracles.
We classify each constraint in the model of the case study system based upon the location(s) of the referenced system data (i.e. the location(s) of the constrained attributes within the model). There are two distinct categories for the constraints: (1) those involving only input and configuration data (the constraint of Fig. 3.4 is in this category), and (2) those involving input, configuration and output data (the constraint of Fig. 3.5 is in this category). The constraints of the first category are used for test validation, while those of the second category are used for the oracle. For each of the categories, we report the number of constraints, the total number of clauses in all constraints, the number of
3.6. Empirical Evaluation
Table 3.2. Size of the input, configuration and output models that were created for the case study system.
File Classes Attributes Associations Generalisations
Input 36 156 17 4
Configuration 9 30 6 1
Output 23 132 15 0
Total 68 318 38 5
Table 3.3. Information about constraints for the case study system classified by the files to which they apply.
# of # of # of Operations # of Iterative
File Constraints Clauses on Collections Operations
Input/Configuration 27 84 20 7
Input/Configuration and Output 22 125 17 29
Total 49 209 37 36
operations on collections (e.g. indexO f ) and the number of iterative operations (e.g. f orAll). The results show (in Table 3.3) that the number of constraints is similar for the two categories of constraints (27 and 22). The results also show that the complexity, represented by the total number of clauses and iterative operations in constraints, mainly lies in the constraints that are used as the oracle (125 clauses compared to 84 and 29 iterative operations compared to 7).
Iqbal et al. show that they can obtain models of similar sizes for similar systems [Iqbal et al., 2012]. More specifically, they created models using UML extended with the Modeling and Analysis of Real-Time Embedded Systems (MARTE) profile to conduct four industrial case studies. Due to the differences in the modelling methodologies and on the problems addressed and the domains, we cannot directly compare the model compositions and sizes to that of this study. However, considering the size metrics reported gives us confidence that the size of our model and constraints is reasonable. One of their case studies used modelling notations similar to our own; the related model contained a total of 71 classes and 16 OCL constraints (as well as other elements). Similarly, Sabetzadeh et al. use SysML to model a real safety-critical SW/HW interface [Sabetzadeh et al., 2011]. Their design consisted of 194 elements having 186 relations and 57 attributes.
Answer to RQ:
The results show that the size of the model and constraints is reasonable compared to typical system model sizes. The data model created for this empirical study contained a total of 68 classes, 318 attributes, and 49 OCL constraints. Most significantly, the cost of modelling was considered acceptable by the SES-DAQ system’s engineers, especially compared to the benefit of defining the constraints used for test validation and oracles.
3.6.4
Threats to validity
In this section, we discuss the threats to validity in this study following the standard classification of threats [Wohlin et al., 2000].
Chapter 3. Modelling Methodology
External threats
The external threats are related to the choice of case study system and the ability to generalise results. Although we only used one system in the empirical evaluation, it is representative of data processing systems. Our results might only be relevant in this application domain; nevertheless, this domain is important and widely used. Many other types of data processing systems have similar characteristics as they process very complex inputs, detect input errors, and extract data. In other words, our ap- proach might also be applicable to any system where the complexity lies in the input, output and their mappings.
Construct threats
For studying scalability, we used the size of the model and constraints. As we mentioned before, we chose the size of the model and constraints instead of modelling effort because the latter can vary greatly based on the modeller’s expertise. We reported all elements of the model and constraints that can be counted. Such data, which is representative of what can be approximately expected in practice, can then be used in context to estimate modelling effort.