Modelling and simulation development process

An example of a model as the output of the development process comes from the field of modelling and simulation where model development processes are used for model creation and maintenance in a more structured and easily controllable and reproducible manner [11, 121].

5.2.2.1 Discrete-event system simulation process [12]

An example of a development process from the field of modelling and simulation is that for discrete-event systems simulation [12, p. 34]. Such simulations deal with systems where the state variable changes at a discrete set of points in time. Banks et al. [12] propose a simulation study lifecycle for such systems that is composed of 12 steps. The process can be seen in Fig. 5.4. The first step is the problem formulation, where the problem to be solved is stated and where possible conflicts in the problem statement are resolved. The second step is the setting

of the objectives and the overall project plan, where the questions to be answered are defined,

and the plans of the study is made in terms of participants involved, cost of the study, number of days required for accomplishing each phase of the project, and the results expected at the end of each phase. The third step is the model conceptualisation which deals with the construction

3UREOHPIRUPXODWLRQ 6HWWLQJRIREMHFWLYHVDQGRYHUDOOSURMHFWSODQ 0RGHO FRQFHSWXDOLVDWLRQ 'DWDFROOHFWLRQ 0RGHOWUDQVODWLRQ 9HULI HG" 9DOLGDWHG" ([SHULPHQWDOGHVLJQ 3URGXFWLRQUXQVDQGDQDO\VLV 0RUH UXQV" 'RFXPHQWDWLRQDQGUHSRUWLQJ ,PSOHPHQWDWLRQ 1R <HV 1R 1R <HV <HV <HV 1R            

Figure 5.4: The discrete-event system simulation process (Figure adapted from [12, p. 35].)

of the model. Banks et al. [12, p. 36] advise the developer to start with a simple model and then build toward greater complexity. They also advise the participation of the model user in the model conceptualisation as it will enhance the quality of the resulting model and increase the confidence of the user.

The fourth step is the data collection, which deals with gathering the needed input data. As the complexity of the model changes, also the required data elements may change. The fifth step is the model translation where the conceptual model is implemented in a computer- recognisable format. The sixth step is the model verification. It involves checking whether the implemented model is performing properly, namely whether the input parameters and logical structures of the model are correctly represented. This step is followed by the model vali-

dation where the model is calibrated by iteratively comparing the model behaviour with the

actual system behaviour. The model is then iteratively improved until a satisfiable accuracy is achieved. The eighth step is the experimental design where the alternatives to be simulated are defined. For each system design that is to be simulated, the length of the runs, the number of replications, and the initialisation period are decided. The ninth step is the production runs

and analysis which estimate the measures of performance for the system designs that are being

simulated. The tenth step decides whether more runs are needed and what design they should have. The eleventh step is the documentation and reporting which is divided into program doc- umentation and progress documentation. While the first documents how the program operates, the latter provides information about the simulation project history. The last step is the imple-

mentation where based on the simulated solutions and their performance, the client makes the

decision about the solution to be implemented.

although it cannot be directly applied to activity recognition in its current form. The reason for that is that while the output of the simulation process is information about which problem solution would be most appropriate, in the case of activity recognition, the output is that of the implemented system, and the question then is not only what is the best way of predicting the user actions but also how well it is able to predict the user actions. Also in the case of activity recognition, we cannot change the available data so that it will represent the needs of the model, but rather the other way around – the model has to be able to represent and reason about the available data. Furthermore, the process is designed for a setup where the actual data is recorded after the simulation model is developed and evaluated. In the field of activity recognition it is often the case that the data is recorded beforehand and due to the cost of the experiment (in terms of participants, settings, infrastructure etc.) one is unable to repeat the experiment. It is also the case that the model is built in order to explain the data, and not vice versa2. Another aspect that simulation processes do not cover is that an activity recognition model has to cope with future data, so it cannot rely only on learning from the available data. For that specific reason prior knowledge is utilised in the activity recognition models.

5.2.2.2 A systematic methodology for developing discrete event simulation models [121]

An alternative methodology for developing discrete event simulation models was proposed by Rus et al. [121]. It has five separate phases and could be considered as an adaptation of the waterfall model. Fig. 5.5 shows the process phases and the steps of each subphase. The first phase involves the simulator requirements identification and specification where the

5HTXLUHPHQWVLGHQWLI FDWLRQ DQGVSHFLI FDWLRQ 3URFHVVDQDO\VLVDQG VSHFLI FDWLRQ 'HVLJQ ,PSOHPHQWDWLRQ 9DOLGDWLRQDQGYHULI FDWLRQ 'HI QLWLRQRIJRDOVDQG TXHVWLRQV 'HI QLWLRQRIXVDJH VFHQDULRV 'HYHORSPHQWRIWHVW FDVHV 9DOLGDWLRQRI UHTXLUHPHQWV $QDO\VLVDQGFUHDWLRQRI SURFHVVPRGHO 5HODWLRQVEHWZHHQ SDUDPHWHUV 'DWDFROOHFWLRQDQG DQDO\VLV 4XDQWLI FDWLRQRI UHODWLRQV +LJKOHYHOGHVLJQ 'HWDLOHGGHVLJQ

Figure 5.5: The discrete-event system simulation process proposed by Rus et al. [121].

purpose and the usage of the model are defined. Also the questions the model has to answer are determined as well as the data needed to answer these questions. The phase is divided into four steps: the first is the definition of goals, questions, and the necessary metrics for answering the questions; the second is the definition of the usage scenarios where the use cases are defined that are used for answering the problem questions. The third step is the development of the test cases which are used to verify and validate the model and the resulting simulation. The last step is the validation of the requirements, where the customer involved in the activity must agree with the content of the model specifications.

The second phase is the process analysis and specification which involves the understand- ing, specification and analysis of the process that is to be modelled. This phase is also divided into four steps. The first is the analysis and creation of a static process model which means that the problem is analysed, an abstract model is defined and later refined. The process model 2_{This brings an interesting discussion about combining simulation processes and activity recognition. One}

could produce simulated data on which the model is validated, and afterwards to conduct the recording of a real data for the model evaluation.

then shows which activities transform which artefacts and how information flows through the model. The second step is the creation of the influence diagram for describing the relations between parameters of the process. Here influence factors are such that change the result or behaviour of other project parameters. The third step is the collection and analysis of empirical data for deriving the quantitative relationships between process parameters. The last step in this phase is the quantification of the relations and the distinguishing of the different parameter types. The output of the second phase are models of the software development process and parameters that have to be simulated, as well as the description of the assumptions that were made during the phase.

In the third phase is the model design is developed that is independent of the model im- plementation. The design is divided into two parts: high level design and detailed design. In the high level design the surrounding infrastructure is defined, as well as the way in which the input and output data is managed and represented. In the detailed design the decisions about which activities to be modelled are made, as well as the items that are to be represented, and the attributes they should possess. Finally, the flow of the different items is defined.

The fourth phase is the model implementation, where all the design decisions and the nec- essary information are transferred into a simulation model. The last phase is the validation and verification, where it is proved whether the model is suitable for the problems it should address. This development process, similarly to the previous simulation process, cannot be directly applied to an activity recognition problem. This is due to the fact that while simulation deals with the problem of determining what will be the most suitable solution to a problem, model- based activity recognition already provides a model that is a solution of the problem, and the question then is ”how well the model performs”. Additionally, the fact that the early phases are no longer revisited at a later stage, means that problems discovered during the implementation and validation phases cannot be fixed by returning to the early stages.

5.2.2.3 A life cycle for modelling and simulation [11]

Another modelling and simulation development process is the life cycle proposed by Balci [11]. In it the development process is viewed from four different perspectives – process, prod- uct, people, and project. The life cycle specifies the work products that are to be created under the corresponding processes, together with the integrated verification and validation activities. It also structures the development and provides guidelines for project management, and finally, identifies areas of expertise in which to employ qualified people. Fig. 5.6 shows the phases in the proposed model. Although the arrows are sequential, the process is not intended to be linear or sequential. It is rather an iterative process where reverse transitions are expected.

The first phase in the development process is the problem formulation where the problem at hand is systematically analysed executing the following steps: establish the problem domain boundary; gather data and information about the problem; identify the stakeholders and deci- sion makers; specify the needs and objectives of the stakeholders and decision makers; identify and specify the constraints; and clearly specify all assumption made so far. The second phase is the requirements engineering which deals with identification and specification of requirements based on the formulated problem. Balci argues that a use case based requirements engineering is the best practice for requirements identification and specification, because a use case repre- sents a small amount of the work the model is supposed to perform, thus it allows decomposing the complex problem into subproblems. The third phase is the conceptual modelling which deals with developing the highest layer of representation of conceptual constructs and knowl- edge. The conceptual model is created to assist in designing many different types of simulation

3UREOHP IRUPXODWLRQ 5HTXLUHPHQWV HQJLQHHULQJ &RQFHSWXDO PRGHOOLQJ $UFKLWHFWLQJ 'HVLJQ ,PSOHPHQWDWLRQ ,QWHJUDWLRQ ([SHULPHQ WDWLRQ 3UHVHQWDWLRQ &HUWLI FDWLRQ 5HSRVLWRU\ RIFHUWLI HG PRGHOV

Figure 5.6: The life cycle for modelling and simulation proposed by Balci et al. [11] (Figure adapted from [11].).

models, thus creating model reusability on a conceptual level. The fourth phase is the archi-

tecting. It deals with specifying the model architecture based on the conceptual model. Balci

explains that there is a distinct difference between design and architecture, as a design is an in- stantiation of an architecture. This is also the next phase in the development process where the model design is derived from the architecture specification. Balci proposes decomposition and modularisation as solutions to reducing and managing the design complexity. The sixth phase is the model implementation, which takes the model design from the previous phase and programs it using a simulation software product. To overcome the model complexity, it is decomposed into submodels. The seventh phase is the model integration which deals with combining the individually developed submodels into an integrated simulation model. The eight phase is the

experimentation, exercise or use. During this phase the model is experimented with. This phase

produces the simulation results based on which a solution for the problem could be chosen. The ninth phase is the model presentation which involves the interpretation and documentation of the simulation results, and their presentation to the decision makers. Aside from these nine phases there is the model certification that awards a certificate that the model satisfies a spe- cific quality criteria. Balci’s life cycle also regards the model storage together with all the documentation and data in a repository that allows future reuse.

Although more complex than the process proposed by Rus et al. [121] it basically has the same structure and limitation. To be applied to the field of activity recognition such model should be adjusted to the specific need of this field. The biggest drawback in it is that while it provides a basis for choosing the best problem solution, an activity recognition model is already a problem solution, so the choice of what and how to be modelled should be made already in the first phases of the process.

The three presented methodologies could all be fitted in a general model of a simulation experiment. Leye et al. discuss different modelling and simulations life cycles and propose a general structure for modelling and simulation experiments [88]. It contains six phases – specification where the experiment is defined, configuration where the model parameters are

selected, simulation where the model is executed, data collection where simulated data of in- terest is collected, and evaluation where the model results are assessed. These phases resemble parts of the intuitive process from Chapter 3 – the model implementation and evaluation where the model is applied to the available data and the estimated behaviour is analysed and evaluated. However, these are just the later parts of the activity recognition model development. Before that the test data has to be collected, annotated, and the model itself has to be developed. Fur- thermore, development processes do not concern themselves with the problem of developing the probabilistic model structure that is responsible for providing adequate activity recognition.