Software process simulation modelling: A survey
of practice
R Ahmed*,1, T Hall2, P Wernick2, S Robinson3and M Shah4
1University of Teesside, Middlesbrough, UK;2University of Hertfordshire, Hatfield, UK;3University of Warwick, Warwick Business School, Coventry, UK;4Cranfield University, Cranfield School of Management, Bedford, UK
In recent years, simulation modelling of software development processes has attracted considerable interest in software engineering. Despite the growing interest, there is little literature available that reports on the state-of-practice in software process simulation modelling (SPSM). We report results of a survey of simulation in SPSM and relate it to simulation practice in general. The results of this survey indicate that software process simulation (SPS) modellers are generally methodical, work on large complex problems, develop large models, and have a systematic simulation modelling process in place. However, on the other hand, the simulation modelling process and simulation model evaluation have been identified as the most urgent problems to be addressed in SPSM. The results from this investigation are interesting and bring many problems into focus. The paper helps understand the characteristics of the SPSM and SPS modellers, and highlights areas of interest for further in-depth research in the SPSM.
Journal of Simulation(2008)2,91–102. doi:10.1057/jos.2008.1
Keywords:software process simulation modelling; survey; simulation practice; modelling process
1. Introduction
This paper aims to identify the practice of software process simulation (SPS) to the wider simulation community and discuss how software process simulation modelling (SPSM) practice compares with simulation modelling in general. We report the results of a survey investigating current state-of-practice in SPSM. This survey explores the simulation practices of expert simulation modellers and identifies the problems believed by the SPSM community to be significant. The results presented represent an up-to-date picture of the use of simulation in the domain of SPSM.
Simulation modelling has developed relatively recently as a discipline in software engineering. Since 1989 when Abdel-Hamid (1989) developed a simulation model for a software development process, there has been an increasing degree of interest in simulation modelling to study, control, manage, and improve software development processes. SPS modellers have come together to form an SPSM community. The SPSM literature largely consists of experience reports and success stories of applying simulation in various areas of software engineering. However, little effort has been put into understanding the characteristics of the SPSM community and their simulation practices in general. To make advances in a field, it is important to understand the characteristics of that field (2000) so that deficiencies can be identified and arenas for improvement determined. Surveys are a com-monly used research tool to explore the characteristics and
trends in a population. Our survey was intended to elicit information to help answer the following research question:
What is the current state-of-practice in SPSM?
To answer the research question, this survey explores the characteristics of SPS modellers, the kind of models they develop, the way they develop their models, their modelling process, and the problems faced by the people in SPSM. The survey was built on the themes identified from the literature. A questionnaire that explored current activities in SPSM was constructed. This survey also replicates some parts of a survey conducted by Willemain (1994) with modellers in operational research (OR).
The paper has been organized in five sections. The paper introduces a discussion on use of surveys in simulation research in Section 2. Section 3 briefly describes SPSM. Section 4 presents the research methodology. Section 5 presents the results of the survey. Section 6 provides an analysis and discussion of the results. Section 7 concludes the paper and outlines future research.
2. Surveys in simulation research
No surveys of simulation practice have been reported in the SPSM literature except a short report of our survey at the 5thInternational Workshop on Software Process Simulation and Modelling (Ahmedet al, 2004a). Meanwhile, the general simulation literature reports several successful studies published in various conferences and journals. Within simulation modelling, the surveys have been used to explore the characteristics of simulation modellers, simulation *Correspondence: R Ahmed, School of Science and Technology,
University of Teesside, Middlesbrough TS1 3BA, UK. E-mail: [email protected]
models, users, simulation tools, the nature of simulation projects, and simulation practice in general. Surveys in simulation research have been conducted for two main purposes:
To explore the current state-of-the-art
To determine how the current state-of-the-art can be improved
Similarly, our survey aims to explore and understand the current state-of-practice in SPSM with a view to identifying areas where this could be improved.
2.1. Some surveys in OR modelling and simulation One of the prominent surveys of OR modelling is by Willemain (1994). This was conducted as part of a larger study that explored the way expert modellers develop their models. Willemain (1994) studies the practices of expert modellers and suggests that practical guidelines for model formulation should be developed for students in order to become experts. Mela˜o and Pidd (2003) report a survey of the practices of discrete event simulation (DES) modellers who develop simulations for business processes. They develop a taxonomy of concepts held by the modellers about business process simulation. Hollocks (2001) reports a survey of user practice in experimentation for discrete event models. He identifies particular areas of concern and identifies deficiencies in simulation tools for experimentation processes. Cochranet al(1995) survey the characteristics of simulation projects in industrial settings and suggest improvements in this area of simulation practice. Hlupic conducted a study of simulation software from a user perspective. Murphy and Perera (2001) highlight the difference in approaches taken by large companies in the UK and USA to implement simulation models in their organizations and explain why these differences exist. Pollacia (2004) conducted an investigation of the state-of-the-art in DES languages. This survey helped identify future trends in simulation languages and tools. Terzi and Cavalieri (2004) conducted a survey of simulation practice for the problems of supply chain management. Kleijnen (2005) conducted a survey on use of simulation tools and techniques in supply chain management. More recently, Taylor and Robinson (2006) debate on future of discrete simulation based on results from a survey of editorial board ofJournal of Simulation.
2.2. Role of surveys in simulation practice
Arguably, the studies mentioned in the last section have helped researchers understand the characteristics of simula-tion modellers and simulasimula-tion practice; and predict and improve the simulation practice and simulation tools and the
way simulation is used in the industry. SPSM researchers can be criticized for eagerly prescribing simulation modelling as a tool to help understand, manage, and control software development processes, while not being aware of what practitioners might actually be doing.
We believe that investigating the current state-of-practice in SPSM will enable an understanding of the characteristics of SPSM and the SPSM community. It will also help develop areas of research for improving simulation methodology in SPSM. Paulk et al(2000) suggest that surveys are a highly useful means of getting a feel for the state-of-the-art and practice in a particular discipline. The survey that we have therefore conducted highlights the characteristics and practices of SPS modellers, helps characterize their simula-tion modelling process, and identifies issues in SPSM that need to be addressed.
3. Software process simulation modelling
Growing competition in the software industry has resulted in increasingly complex software processes to address issues such as quality, cost, and development time. Software (development) processes are a collection of various activities, for example, cost estimation, size estimation, requirement specification and analysis, initial design, detailed design, implementation, code inspections, and testing, etc. All these activities are dynamically tied together in complex relation-ships. On the other hand, high personnel turn over, rapid development requirements, frequent changes in software requirements pose complex challenges in management of the development process. Capturing and forecasting the effects of these complex dynamic and stochastic relationships early in a project may allow better planning and project control. Otherwise unanticipated process behaviour at one process step may ripple through the whole lifecycle of the project. Therefore, simulation modelling of software pro-cesses has been promoted as a tool to understand, study, and manage software development processes (Mu¨nch and Pfahl, 2006).
An SPSM is used to study some particular software activity, such as development, maintenance, or evolution (Kellneret al, 1999). Studies show that simulation modelling has proved to be an effective tool to study organizational software development processes and forecast potential change and improvement in those processes (Raffo, 1998). Kellner et al (1999) suggest that simulation modelling of software processes is particularly desirable when:
Complexity is beyond human intuition.
There is uncertainty and stochasticity in system behaviour.
Risks in process change are very high.
The system has some dynamic behaviour.
Decisions made at one point in the system may impact on the process in other aspects.
Although simulation modelling has been applied very commonly in other scientific and business processes (eg defence, air traffic control, demand and supply chains), it is relatively new to software engineering practices.
In SPSM, system dynamics (SD) and DES are the most commonly used techniques. Moreover, hybrid simulation (SD and DES combined) has also been applied to simulate software processes (Martin and Raffo, 2000).
A discrete event model allows us to represent the sequential interdependence that occurs between activities in a software process. Activities in a development process, for example, may be delayed when a programmer is diverted to another task. Testing may be delayed until a test bed is released. A DES model captures these sequentially depen-dent activities and provides ways to analyse the results of various what-if scenarios. Figure 1 shows a DES model of a software review process (Huff, 1996) and represents the
sequential activities. Ten items are waiting to be reviewed; each item goes through the review process sequentially.
In continuous models, the physical or logical entities moving around or stored in the process by which the system is developed are represented by continuous quantities that can change at every step of simulated time. The structure of the system is typically shown by causal feedback loops, and system state is shown by flows and level variable values. Abdel-Hamid (1989) applied SD to software projects for the first time. Later work by Madachy (1994) modelled a more detailed development process. Tvedt and Collefello (1995) used SD to model the software inspection process.
The FEAST (Feedback, Evolution And Software Tech-nology) project at Imperial College, London has been an influential simulation study for software processes. Figure 2 shows a stock-flow diagram representing a software
Figure 1 A DES model of software review process (Huff, 1996).
Approval for Release or Rework
STAFFING POLICY
Released System
Anti Regressive Deficit Growth Productivity <Time> PROGRESSIVE ANTI-REGRESSIVE RATIO Work Awaiting Assignment TIME STEP BASELINE PRODUCTIVITY Assigned Work Completed Cumulative Anti Regressive Work Outputs Implementation Flow <Time> Adaptation Work Requests Rework Validated Components Component Validation Work Implemented Integration Flow TEAM SIZE Validated Elements System Validation New Work Requests
Anti regressive effort Rejected Work Requests Discarded StandBy Progressive effort Cumulative Progressive Work RELEASE POLICY Flow to StandBy Delayed Decision Release Rate
Anti regressive work DOMAIN FACTORS Impact of Anti Regressive Deficit <Time> Effective Productivity
evolution process (Lehman and Ramil, 1999) developed in the FEAST project. It shows that various variables are connected to each other through complex causal feedback loops. In real software process, these complex feedback loops influence the software process significantly and it is desirable to understand and take account of these effects in advance.
A range of simulation studies can be found in the literature applied to different areas of software engineering. Kellner et al (1999) categorizes the particular areas of software engineering in which simulation modelling has been applied. This includes strategic management of software processes (Henderson and Howard, 2000; Roehling et al, 2000), software project planning (Abdel-Hamid, 1989), control and operational management of software project/ process (Martin and Raffo, 2000), software process im-provement and technology adoption (Scacchi, 2000), under-standing of software processes (Misic et al, 2004), and training and learning in software engineering (Navarro and Hoek, 2005).
4. The survey
To understand the state-of-practice of SPSM, the survey has been conducted among practitioners in SPSM who develop models in research and/or industrial contexts. The ques-tionnaire that we developed is divided into six parts as follows:
1. Describe yourself as a modeller 2. Describe the models you develop 3. Describe the problems you model
4. Describe the most typical way you develop your models 5. Describe your modelling process
6. Describe the critical problems in SPSM
The first four sections of the questionnaire largely replicate the questionnaire survey conducted by Willemain (1994) with 12 expert modellers in the field of OR. We have added a fifth section to investigate what process simulation modellers use for developing models of software processes and a sixth section to identify the most critical problems in SPSM.
4.1. Survey respondents
The population for this survey is SPSM practitioners and researchers. It is a difficult-to-find and difficult-to-access population because no sampling frame exists for it and the population is dispersed across the world. This survey was conducted at the International Workshop on Software Process Simulation & Modelling 2003 (ProSim’03) May, 2003 held at Portland, Oregon, USA, which is the major annual forum for SPS modellers. The ProSim series of workshops attracts an international body of simulation
modellers from both industry and academia each year; the organizers and participants include leading workers in the field of SPSM. Each year a growing number of SPS modellers attend ProSim. Consequently, we chose the ProSim’03 participants as a sample of SPSM modellers to whom we addressed our questionnaire.
One can argue that these active researchers are not a representative sample of SPS modellers. There is no way to guarantee the representativeness of a non-random sample (Oppenheim, 1992). However, it is worth noting that simulation is a relatively new discipline in software engineering. Hence, although there is an increasing interest in simulation of software development processes, the SPS community is still quite small. Therefore, this is probably the best sample size we could get for a survey of practice in SPS. Nevertheless, our respondents are among the people who are defining the future of SPSM by their contributions to research and practice. We feel that their views should therefore be considered important in developing an under-standing of, and attempting to improve, SPS practice. Since this is an exploratory study, we do not aim to derive concrete conclusions from a relatively small sample. This study may be used as a starting point to conduct further research into various problems in SPSM.
Questionnaires were distributed to all 30 delegates at the workshop. To ensure a higher response rate the question-naire was distributed and collected by hand. Seventeen responses were obtained, representing a response rate of 57%; that this is five more participants than Willemain’s original study (1994).
Among the 17 respondents nine are academics (one research student), and eight respondents are from industry. Hence, the sample is a good mix of academics and practitioners. Figure 3 shows the experience profile of the respondents in simulation modelling. The average simulation modelling experience of the sample is 6.5 years.
4.2. Questionnaire design
The questionnaire consists of three types of questions: classification, behavioural, and attitudinal. Classification questions aim to discover demographic information about the respondents such as the nature of their job, experience, and types of simulation model they develop. Behavioural questions aim to establish what respondents do, that is, their simulation modelling practices. Attitudinal questions aim to uncover their perceptions about the critical problems in simulation modelling. Most of the questions in the questionnaire are closed and required specific information from the respondents.
The behavioural part of the questionnaire is constructed on 6-point Likert scales. The original questionnaire by Willemain (1994) consisted of 7-point Likert scales. Willemain’s (1994) survey results show a tendency to choose a middle value. We converted this into a 6-point scale to
prevent respondents from choosing a middle value. There is a counter argument that if we do not give a middle value choice, it means we are pushing the respondents to give their opinion while someone may have a neutral perception. Researchers have varying opinions about the number of choices for Likert scales. Likert himself, in his original paper, did not consider the number of choices to be an important issue (Likert, 1932).
The attitudinal part of the questionnaire provides the respondents with a set of choices for each question. These choices were identified from the simulation literature. The respondents also had the option to specify an alternative answer.
5. Survey results
The results are summarized into six tables corresponding to the six parts of the questionnaire. The response distributions have been shown in the tables for each part separately. We also compare the results of the first four parts of the questionnaire with Willemain’s (1994) results.
Following Willemain (1994), we consider a consensus in response if 75% of the responses lie on one side of the scale. Owing to the size of the sample we do not attempt to generalize the results to a population by showing a consensus, rather highlight general trends in the responses. The items with consensus have been marked in bold face.
5.1. Part I—Describe yourself as a modeller
Part I of the questionnaire asks respondents about the kind of modeller they are. Table 1 shows the response distribu-tions. An interesting profile of modellers emerges in this section. Most respondents consider themselves as applied modellers, who value comprehensiveness in their models, and who are methodical rather than intuitive, valuing practicality rather than creativity. This implies that most of the respondents work on applied problems, where the emphasis is to provide a comprehensive solution to the customer following a methodical way.
Willemain’s (1994) survey of 12 modelling experts in the field of OR did not generate consensus on any of the items in Part I. Comparing with Willemain (1994) results, it shows that respondents in Willemain’s survey have a tendency to choose the middle value or most responses are evenly distributed on two sides of the scales. However, in our results, Table 1 shows that most responses lie on one side of the scale. Therefore, we get consensus on most of the questions in Part I of the questionnaire.
It is interesting that all the participants in Willemain (1994) were practitioners whereas our sample is a mix of practitioners and researchers. One might expect the practi-tioners to be more applied and methodical in their simulation modelling than researchers. Nevertheless, our sample, which is a mix of practitioners and researchers, appears to be more applied and methodical in their simulation modelling. This indicates that there might be some fundamental differences in SPS modellers and OR modellers. While half of our respondents are researchers, the majority claim to be applied modellers. This suggests that most of the ‘researchers’ in our sample might be more accurately described in our terms as applied modellers because they conduct simulation studies in collaboration with industry where their simulation models are applied. However, this raises the question of why Willemain (1994) fails to find consensus on these questions when all of his respondents were practitioners working in industry.
The simulation literature suggests that it is important to be methodical and practical, as well as intuitive and creative, in simulation modelling. Shannon (1975) states that simula-tion modelling is both art and science, and that a simulasimula-tion study has therefore to be undertaken in a manner that is both methodical and creative in order to produce a good
Table 1 Describe yourself as a modeller
Q # 1 2 3 4 5 6
1.1 Theoretical 0 0 3 7 7 0 Applied
1.2 Value elegance 1 0 2 9 5 0 Value comprehensiveness
1.3 Intuitive 0 0 3 4 9 1 Methodical
1.4 Value creativity 1 1 1 8 5 1 Value practicality
1.5 Generalist WRT subject area 1 1 3 8 2 2 Specialist
0 0.5 1 1.5 2 2.5 3 3.5 4 No. of participants 1-4 5-8 9-12 Experience (years) Academic Practitioner
Figure 3 Respondents’ experience profile in simulation mod-elling.
model. This may mean that respondents in our survey take simulation modelling more as a science than art because of their background in software engineering, therefore they tend to be more methodical and practical, and less intuitive and creative. Meanwhile, Willemain (1994) respondents tend to choose middle value because they want to be both methodical and intuitive, and practical and creative.
5.2. Part II—Describe the models you develop
Part II of the questionnaire asks the respondents about the models they develop. Table 2 shows the response distribu-tions. Consensus has been obtained only on two questions, that is, the models they develop are dynamic in behaviour (Q2.4). Secondly, most of them develop probabilistic models (Q2.9) that involve prediction of situations in advance. This may be related to Q2.7 where most of the respondents develop simulation models that are data based.
More than half of the respondents use their models themselves as against handing them over to ‘end users’. This number includes six respondents from academia and three respondents from industry. Responses to the rest of the questions are mixed and distributed across the six scales. However, a tendency of choosing middle values is observable. In this part, Willemain (1994) obtained consensus only on Q2.1. Similarly we obtain consensus only on two questions. The results for this part both in Willemain (1994) and our survey show a tendency to choose middle value and lower level of consensus. This suggests that there is a high level of diversity in the kind of models developed by both Willemain’s and our respondents.
5.3. Part III—Describe the problems you model
Table 3 summarizes the results for Part III of the questionnaire. This part presents the kind of problems respondents set out to model. We found consensus on every question in this part of the questionnaire. Table 3 shows that most of the respondents model a system that exists, have specific objectives for the problem in hand (only one respondent does not have specific objectives), model complex systems, work on large problems, solve problems that have immediate relevance, and model process-related problems.
Willemain (1994) obtained consensus only on the first three questions in Part III. We are unable to give a definite explanation as to the difference in our responses. We assume the difference in responses in the two surveys is because the respondents are from two different disciplines. As a result, they have different perceptions and different types of problems to work on. The perceptions of size and complex-ity of problem would vary by each individual. For example, it is not clear from survey data what the size of problems are when SPSM modellers say that they work on large problem. It may also be that the problems are perceived to be large because they tend to model to a very low level of abstraction.
5.4. Part IV—Describe the most typical way you develop your models
Part IV of the questionnaire examines respondents’ typical way of modelling. The results are shown in Table 4. Table 4 shows that most of the respondents look for analogies to solve the problem, start with a small amount of content and Table 2 Describe the models you develop
Q # 1 2 3 4 5 6
2.1 Each model unique 1 3 5 7 1 0 All basically identical
2.2 Only you use them 2 5 2 1 7 0 Many others use them
2.3 Goal is insight 3 3 5 5 1 0 Goal is quantities
2.4 Dynamic in behaviour 8 3 3 1 1 1 Static
2.5 One unified model 4 1 4 4 1 3 Many linked sub-models
2.6 Small number of variables (say 10–30) 1 2 3 5 4 2 Large number of variables (say 100þ)
2.7 Data based 3 1 7 3 2 1 Theory based
2.8 Descriptive 5 2 4 4 2 0 Prescriptive
2.9 Deterministic 1 1 2 6 2 5 Probabilistic
Table 3 Describe the problems you model
Q # 1 2 3 4 5 6
3.1 System that exists 5 8 2 1 1 0 System only imagined
3.2 Vague objectives 1 0 0 0 11 5 Specific objectives
3.3 Simple systems 0 1 2 3 8 3 Complex systems
3.4 Large problems 2 5 9 0 1 0 Small problems
3.5 Immediate relevance 2 4 7 0 3 1 Long term relevance
then add while developing the model. Also they work over extended periods of time rather than developing the model in one burst. Responses in Table 4 show that most of the respondents have heavy client contact; however, a consensus could not be found. More of the respondents develop a single model rather than developing many alternative models and compare the results.
In this part, we obtain consensus on the first three questions, whereas Willemain (1994) obtained consensus on all questions except for Q4.6. Again this marks a fundamental difference between the respondents in the two surveys. One explanation of this difference might be that modelling is a long-established practice in OR, and so OR modellers as a community are more used to common modelling heuristics and have identified more of them. Meanwhile, given that SPSM is a relatively new discipline, the SPS modellers are not so aware of these common modelling heuristics and have not yet identified which of them might be of use in their specialized discipline. This marks a need for training and guidelines for SPS modellers. However, training and education is the lowest ranked issue on SPS modellers’ list of concerns as shown in Table 6 below.
Responses to Q4.6 indicate that some respondents do not collaborate with others while working on a modelling problem. It is difficult to explain the reason for low collaboration. One potential reason may be that simulation modelling is a new discipline in software engineering. Therefore, organizations do not understand the benefits of simulation and do not provide many resources. Hence very few people work on a simulation project. Moreover, the reason could be that the software industry has relatively few people qualified for this sort of work. Another potential answer can be that for SPSM the nature of the problems typically addressed, or of characteristics of the models developed (size, complexity, etc), or the nature of simulation modelling itself does not demand that many people work together. If this is the case, it would be similar to simulation modelling in OR, where most models are developed by individual modellers (Robinson, 2002). It is interesting to note that Willemain (1994) also did not find a consensus on Q4.6.
5.5. Part V—Describe your modelling process
Part V of the questionnaire explores the underlying process that respondents employ in their modelling practices. Table 5 shows the summary of results. Most of the responses show strong consensus. The majority of the respondents indicate that they consider that they have a systematic process for model development. Other areas of consensus are that as part of their systematic process they assess feasibility as a first step, define the problem sufficiently, always document formally, value a modular structure, value usability, and review the model at each step of their modelling process.
We did not anticipate such a level of consensus in this part, largely because it is rare to find discussions on the simulation modelling process and best simulation practices in SPSM literature from which practitioners could take advice and establish a common core of practice. Therefore, we did not initially expect that SPS modellers would believe that they have such discipline in their simulation practice. This, however, instigates our interest to find out in more depth about their simulation modelling process.
5.6. Part VI—Describe the critical problems in SPSM Part VI of this questionnaire attempts to identify the most critical problems to be addressed by the SPSM community. We show the questions and their responses in Table 5. Respondents have selected multiple answers for these questions.
Table 6 shows that the modelling process and the evaluation of simulation models of software processes are believed by our sample to be the most urgent problems to be addressed. Validation is also proposed as another problem that needs urgent attention. Only one respondent was concerned about modelling notations, which suggests that most of the respondents find the general purpose modelling notations typically employed in SPSM are satisfactory for their work. This in turn may indicate that other features of simulation as undertaken in other disciplines, such as processes, patterns, and heuristics, may also be applicable to SPSM. None of the respondents consider modelling tools and the model development environment as an important Table 4 Describe the typical way you model
Q # 1 2 3 4 5 6
4.1 Look for analogies 3 8 3 2 1 0 Start from scratch
4.2 Start small and add content 6 8 2 0 1 0 Start big and subtract content
4.3 Work in one short burst 0 2 2 5 7 1 Work over extended time
4.4 No client contact 0 2 4 4 4 4 Heavy client contact
4.5 Make single model 3 4 3 3 2 2 Make alternative models and compare results
problem to be addressed. This also implies that respondents are happy with the available tools. Education and training is another aspect that respondents do not consider important. Table 7 shows that modellers are most concerned about model design in the modelling process. It is interesting that Willemain’s (1994) experiments with expert respondents show that they spent 59% of their effort on model structure. This suggests that within the modelling process the problem of model design is in the most need of attention. Model requirements, maintenance, and formal documentation are lower priority issues.
6. Discussion of the results
The results show that the majority of the respondents in this survey are applied and methodical modellers who value practicality and comprehensiveness in their models. They
mostly develop dynamic probabilistic models and mostly work alone, using a systematic process for model develop-ment. Most respondents document their models formally, although it is not clear what the term ‘formal documenta-tion’ means to them and what the value of documentation is to model developers and model users. Most of them also say that they assess feasibility as a first step, define the problem sufficiently, value a modular structure, value usability, and review the model at each step. However, formal model evaluation is not much practiced. About half of the respondents consider maintainability as an important aspect of a model. Most of the respondents work on large and complex problems on which maintainability may be an issue. Some respondents do not have a life cycle perspective on the simulation modelling process. This may be a reason for not giving importance to maintainability. The majority of respondents consider the modelling process and evaluation as the most urgent problems to be addressed in SPSM. Table 6 Most urgent problems to be addressed in SPSM
Issue to be addressed in SPSM Responses
(a) Modelling process 6
(b) Evaluation 6
(c) Validation 5
(d) Modelling notations 1
(e) Modelling tools or model development environments 0
(f) Formal education and training (for model developers) 0
(g) Other (please specify) 3 (Model confidence, data sources and data collection)
Table 7 The area of modelling process needs more attention
Area of simulation modelling process Responses
(a) Model requirements analysis (scope, problem definition, inputs and outputs definition, etc)
3
(b) Model design 7
(c) Formal documentation 3
(d) Maintenance 3
(e) Other (please specify) 6 (Marketing, making simulation modelling accessible
and useful for professionals, verification, reuse, model understanding, application and use)
Table 5 Describe your modelling process
Q # 1 2 3 4 5 6
5.1 Systematic process 4 7 2 1 3 0 Ad hocprocess
5.2 Assess feasibility as a first step 1 9 2 3 1 1 Don’t assess feasibility
5.3 Define model scope at all levels 4 6 1 3 2 1 Scoping isn’t a big concern
5.4 Define problem sufficiently 5 9 1 0 1 1 Insufficient problem definition
5.5 Always document formally 3 4 7 1 2 0 Never document formally
5.6 Follow a life cycle approach 3 7 1 3 2 1 No life cycle approach
5.7 Value modular structure 8 8 0 1 0 0 Develop model in one view
5.8 Concern usability 4 7 3 2 0 1 Usability isn’t of much concern
5.9 Review model at each step 4 5 4 1 2 1 Review at the end
5.10 Evaluate formally 3 4 4 3 1 2 No formal evaluation
6.1. Simulation model size and complexity
Our results show that more than half of the respondents say they develop large models. This can be related to the fact that most modellers consider themselves to be very methodical and they work on large models. These findings may be connected in some way: for instance, are they, or do they need to be, methodical because they develop large models?
Our results also show that most of the respondents develop simulation models of complex systems and most also model large problems. This can be related to the finding that most of them state that they employ a systematic and disciplined simulation modelling process. We are not sure whether they have a systematic process because they develop large complex models or that they naturally work like this. Simulation model size and complexity may have a relation-ship to the way people go about developing simulation models. Current SPSM literature does not discuss such aspects of simulation. The question is seldom asked as to whether simulation model size and complexity have any effect on cost or time, or on the way people go about developing simulation models.
Few discussions of simulation model size and complexity can be found in the general simulation literature (Law, 1998; Arthuret al, 1999; Pageet al, 1999). To date, no substantial study has been published that explores the relationship between simulation model size/complexity and the simulation modelling process as a whole. Arthuret al (1999), however, discuss the effects of model size and complexity in the way verification and validation is performed. They note that with the increasing tool support and computing power, simulation model size and complexity continues to grow. This phenom-enon of growth brings many new problems along with it. Some authors have attempted to draw attention to these issues. In the words of Chwifet al(2000), ‘nowadays size and complexity of models is growing more and more, forcing modellers to face some problems that they were not accustomed to’. Pageet al(1999) outline some of the problem posed by the increasing scale of simulation models:
How a large model should be expressed
How to validate a large model
How to solve and get answers from a large model
How to trace and understand the output of a large model The SPSM community also needs to pay attention to the problems of simulation model size and complexity and the way it affects their simulation model development and validation practices.
6.2. Simulation modelling process
Although the SPSM literature lacks studies related to the simulation modelling process, responses from most of our
respondents suggest that they have a well-defined modelling process in use for simulation model development. Our results also show that the modelling process is thought by them to be the most urgent problem to be addressed in SPSM. However, it is interesting that while most of our respondents claim that they have a systematic modelling process they still consider the modelling process as being the most important problem. An in-depth exploration of their modelling processes should be a matter of importance to the simulation community.
The SPSM literature does not provide much debate on the process of developing simulation models. It is not clear whether SPS modellers simply do not report on the modelling process, because either they are already using what is to them a satisfactory process, or they are not interested in the modelling process. However, it seems that modellers are much more interested in the end product than the process of creating that product. Ruset al(2003) argue that there is a need to migrate SPSM from craft to engineering by adopting a simulation modelling process for model development. This suggests that some researchers at least believe that the simulation modelling process has not been given sufficient attention.
Since both the SPSM literature and the respondents in this survey suggest that the simulation modelling process is the problem that needs urgent attention, we conclude that the SPSM community needs to come together and discuss how important the simulation modelling process is in SPSM. The questions to be considered may include:
How important is it to understand and reflect on the simulation modelling process?
What are the deficiencies in the SPS modellers’ simulation modelling process?
Can this simulation modelling process be improved?
What management, efficiency, and efficacy benefits might this improvement be expected to bring to simulation practice in SPSM?
6.3. Documentation and maintenance
While our results show that the majority of the respondents claim to document their models formally, this result raises the question of what SPS modellers consider to be formal documentation for their models and what the value of documentation is to them and to users of their models. The majority of our respondents indicate that they work on large and complex problems. We would expect that maintain-ability will inevitably become an issue if these models are to be capable of being evolved so that they remain useful in the long term. Our literature review suggests, however, that the maintainability of models has not been given much attention in the general simulation literature. Meanwhile, in this survey more than half of the sample indicates that they are
concerned about maintainability. This raises the question of the importance of simulation model maintenance and how simulation modellers assure the maintainability of their models.
Somewhat in contradiction to the above, our results show that the survey respondents generally think that maintenance and documentation are low priority issues for further research. One reason for this might be that they do indeed take care of the documentation and maintainability of their models, and hence consider it to be a solved problem. Another potential reason could be that simulation models developed in SPSM are too small (although they say they build large models, we do not know as yet how to determine the size and complexity of models) or large but conceptually too simple to be documented and maintained. Another reason could be that most simulation models may not be used in the long term; therefore, documentation and maintenance are not an issue.
Issues of simulation model documentation and mainte-nance are also seldom discussed in the general simulation literature. Fosset al(1998) say that most simulation models are poorly documented and are therefore rarely reused. The models evolve and are redefined over a period of time, and the managers who use the models may change their minds about priorities or themselves be changed frequently. Foss et alfurther state that poor documentation makes it very hard to maintain the models. However, it is generally believed that reusing simulation models is difficult and less cost effective than building a new one from scratch (Robinson et al, 2004). However, the importance of maintenance and documentation has been highlighted for large scale models by Gass (1987).
6.4. Evaluation
More than half of the respondents say that they conduct formal evaluations of their simulation models. However, they consider evaluation to be one of the most important unresolved problems in SPSM. Similarly model validation is another important problem that needs attention. It is difficult to say what meaning the respondents have taken by evaluation. Gass (1987) suggests that the evaluation of models encompasses both validation and verification activities along with an assessment of the models’ quality, usability, and utility. However, on the basis of an analysis of current literature, it appears that this form of evaluation does not have a formal position in the simulation modelling literature (Ahmed et al, 2004b). The general simulation modelling literature seems to emphasize validation and verification activities and considers this to be equivalent to evaluation. In addition, a small number of relevant studies can be found in the SPSM literature that focus on simulation model validation (Ahmed et al, 2004b). As the respondents in our survey believe that evaluation and validation are two important problems in SPSM that need
attention, more research is needed into the rationale of, and requirements for simulation model evaluation and validation in SPSM.
6.5. Simulation education and training
Our results show that simulation education and training is not seen as an issue by our respondents. This is an interesting observation since simulation modelling is a comparatively young discipline in software engineering, consequently many SPS modellers are new entrants in the field. In the general simulation literature, there is an increasing emphasis on simulation education and training (Waiselet al, 1999) and a significant debate can be found on simulation education and teaching (Paul and Taylor, 1997; Standridge et al, 2005; Spiegelet al, 2006). The Winter Simulation Conference runs a dedicated track on simulation education.
Although there is a body of knowledge available in the general simulation literature to guide and educate novices, the SPSM literature lacks information for novice SPS modellers to improve their simulation practice. It would seem that the SPSM community needs to develop its own material to guide and educate novices entering into the field that supports activities and considerations specific to SPSM within the simulation community.
6.6. Differences between SPSM and simulation in OR Some of the results of our survey are similar to those described for the OR respondents in Willemain’s survey (1999). However, most of them are different. In Willemain (1994), consensus is comparatively lower over personal questions and greater on universal questions such as the general nature of the problems and the way they model. Our results show a relatively high level of consensus both on personal and universal questions. The difference in response might be due to protocol-related issues such as the following:
The inherent difference between the samples in two surveys as they come from two different backgrounds.
The difference in number of points in the Likert scales used (Willemain used seven points and we used six).
The difference in sample size, as we have a slightly bigger sample than Willemain (1994).
As discussed above, the results for the part of the survey that was a replication of Willemain (1994) are very different from those of Willemain. A way to resolve this issue would be to explore whether such differences can be found in groups of OR and SPSM simulation modellers deliberately selected to be comparable within their commu-nities. A study of comparative groups would help identify differences in SPSM and simulation in OR and it would
help to transfer best practice knowledge between the two disciplines.
7. Conclusions and future work
In this paper, we have presented the results of a survey conducted with SPS modellers. Seventeen expert SPS modellers, both from industry and academia, took part in the survey. The survey investigates simulation practice and problems in SPSM by asking the question:
What is the current state-of-practice of simulation practice in SPSM?
An interesting profile of SPS modellers and their practices emerges. The survey results indicate that the majority of the respondents use a systematic process for model develop-ment. But on the other hand, the most highlighted problem in SPSM is the modelling process. The survey results also show that most of the respondents are very methodical, work on large complex problems over an extended period of time, build large models, work alone, and document their models formally. Simulation model design is an important problem that needs attention, as is evaluation. However, to most of the respondents maintainability of models is not an issue. Although the SPS modellers do not seem to follow common modelling heuristics, training and education for SPS modellers do not seem to be an issue for our respondents either.
There are several differences and commonalities identified between the SPS and OR simulation modellers based on the survey results. Studying a comparative group of SPS and OR modellers may allow comparison of practices and transfer of knowledge between them. The survey helps us understand the current state-of-practice in SPSM and determine areas of research for the future.
Our survey inevitably gives only an indication of various practices in the SPSM community. We believe that to develop further understanding of simulation practices in SPSM, both broader and deeper studies are needed. Our examination of the survey results raises many questions, and suggests particular areas of SPSM requiring further explora-tion. These include:
What concepts of size and complexity are held by SPS modellers? How does the size and complexity of their simulation models affect the way they do (or should) go about developing simulation models?
What is the ‘systematic process’ employed by SPS modellers? How do SPSM modellers actually go about developing their simulation models? How do they define ‘improvement’ in their simulation development processes, and how would they want their simulation modelling process to be improved? How would they determine whether a change to the development process actually is an improvement?
What concepts are held about simulation model doc-umentation and maintenance by the SPS modellers? How important is it for them to document and maintain their models and what documentation should they hold and why? How do they actually document and maintain their models, and do they find their current documentation useful?
What concepts are held about evaluation by the SPS modellers? How important is it for them to evaluate their models? How do they evaluate their model?
Is simulation education and training given due attention in SPSM? If not enough importance is attached to it, how might the situation be improved?
How does the simulation practice in software engineering compare with simulation practice in OR? What can SPS modellers learn from the similarities and differences? What can simulation modellers in OR learn from SPS modellers?
These questions form the basis of a research programme intended to improve the effectiveness and relevance of SPSM to software engineering. Given the advantages of SPSM and current diverging trends in software process practice (Mu¨nch and Pfahl, 2006), as exemplified by the growth in use of agile approaches (Highsmith and Cockburn, 2001), such a research programme is becoming increasingly urgent.
References
Abdel-Hamid TK (1989). The dynamics of software project staffing:
A system dynamics based simulation approach. IEEE Trans
Softw Eng15(2): 109–119.
Ahmed R, Hall T, Wernick P and Robinson S (2004a). Simulation
modelling practices of ProSim03 modellers: A survey.
In: Dietmar P, David R, Ioana R, Paul W (eds). Fifth
International Workshop on Software Process Simulation and Modeling, ProSim 2004, Edinburgh, Scotland — Proceedings. Stevenage: IEE Publishing.
Ahmed R, Hall T, Wernick P and Robinson S (2004b). Evaluating the quality of software process simulation models. In: Brailsford SC, Oakshott L, Robinson S and Taylor SJE (eds).
Proceedings of The OR Society Simulation Workshop (SW2004), Birmingham, UK, 23–24 March. Birmingham: Operational Research Society.
Arthur JDet al(1999). Verification and validation (panel session): What impact should project size and complexity have on attendant V&V activities and supporting infrastructure? In: Farrington PA, Nembhard HB, Sturrock DT and Evans GW (eds).Proceedings of the 31st Winter Simulation Conference, Piscataway, NJ: IEE, pp 148–155.
Chwif L, Barretto MRP and Paul RJ (2000). On simulation model complexity. In: Joines JA, Barton RR, Kang K and Fishwick PA (eds).Proceedings of the 32nd Winter Simulation Conference archive, 10–13 December, Piscataway, NJ: IEEE, pp 449–455. Cochran JK, Mckulak GT and Savory PA (1995). Simulation
project characteristics in industrial settings. INTERFACES
25(4): 104–113.
Foss BA, Lohmann B and Marquardt W (1998). A field study of the industrial modeling process.J Process Control8(5/6): 325–338.
Gass SI (1987). Managing the modelling process: A personal reflection.Eur J Oper Res31(11): 1–8.
Henderson P and Howard Y (2000). Simulating a process strategy for large scale software development using systems dynamics.
Softw Process Improv Pract5(2–3): 121–131.
Highsmith J and Cockburn A (2001). Agile software development: The business of innovation.IEEE Comput34(9): 120–127. Hlupic V (1999). Simulation software: What users want?Simulation
73(6): 362–370.
Hollocks BW (2001). Discrete-event simulation: An inquiry into user practice.Simul Pract Theory8(6–7): 451–471.
Huff KE (1996). Process measurement through process modeling and simulation.Proceedings of the 10th International Software Process Workshop, Washington, DC, USA: IEEE Computer Society.
Kellner IM, Madachy R and Raffo D (1999). Software process simulation modelling: Why? What? How?J Syst Softw46(2–3): 91–105.
Kleijnen JPC (2005). Supply chain simulation tools and techniques: A survey.Int J Simul Process Model1(1–2): 2–89.
Law DR (1998). Scalable means more than more: A unifying definition of simulation scalability. In: Medeiros DJ, Watson
EF, Manivannan M, Carson J. (eds). Proceedings of the
30th Winter Simulation Conference, Piscataway, NJ: IEEE, pp 781–788.
Lehman MM and Ramil JF (1999). Modeling process dynamics in software evolution processes—some issues. In: Garlan D and Karmer J (eds).International Conference on Software Engineer-ing,Workshop on Software Change and Evolution, Los Angeles, Piscataway, NJ: IEEE, May.
Likert R (1932). A technique for the measurement of attitudes.Arch Psychol22(140): 44–53.
Madachy R (1994). A software project dynamics model for
process cost, schedule and risk assessment. PhD Dissertation, Department of Industrial and Systems Engineering, University of Southern California, Los Angeles, CA.
Martin RH and Raffo D (2000). A model of software development
process using both continuous and discrete models. Softw
Process Improv Pract, Wiley Inter Science5(2): 147–157. Mela˜o N and Pidd M (2003). Use of business process simulation: A
survey of practitioners.J Opns Res Soc54(1): 2–10.
Misic VB, Gevaert H and Rennie R (2004). Extreme dynamics: Towards a system dynamics model of the extreme programming software development process. 5th International Workshop on Software Process Simulation and Modeling (ProSim 2004)’’ W1 1L Workshop — 26th International Conference on Software Engineering (2004/911), pp. 237–242. Edinburgh, Scotland, UK, 24–25 May 2004.
Mu¨nch J and Pfahl D (2006). Guest Editorials. Special Issues on ProSim 2005.Softw Process Improv Pract11(4): 339–343. Murphy SP and Perera TD (2001). General applications:
Simulation practice: Key enablers in the development of simulation. In: Peters BA, Smith JS, Medeiros DJ and Evans
GW (eds). Proceedings of the 33rd Winter Simulation
Con-ference, Piscataway, NJ: IEEE, pp 1429–1437.
Navarro EO and Hoek A (2005). Software process modeling for an
educational software engineering simulation game. Softw
Process Improv Pract10(3): 311–325.
Oppenheim AN (1992). Questionnaire Design, Interviewing and
Attitude Measurement. Continuum: London.
Page EH et al (1999). Panel: Strategic directions in simulation research. In: Farrington PA, Nembhard HB, Sturrock DT and
Evans GW (eds). Proceedings of 31st Winter Simulation
Conference, Piscataway, NJ: IEEE, pp 1509–1520.
Paulk MC, Goldenson DR and White DM (2000). The 1999 survey of high maturity organizations. Software Engineering Institute, CMU, Pittsburgh CMU/SEI-2000-SR-002, February.
Paul RJ and Taylor SJE (1997). Simulation modelling methodology and education.Informatica21(4): 579–586.
Pollacia FL (2004). A survey of discrete event simulation and stateo-the-art discrete event languages.ACM SIGSIM Simula-tion Digest 200420(3): 8–25.
Raffo M (1998). Software process simulation modeling.(ProSim’98): Workshop Report, Empirical Software Engineering, pp 407–412. Robinson S (2002). Modes of simulation practice: Approaches to
business and military simulation. Simul Model Pract Theory
10(8): 513–523.
Robinson S et al (2004). Simulation model reuse: Definitions,
benefits and obstacles. Simul Model Pract Theory 12(7–8):
479–494.
Roehling ST, Collofell JS, Herman BG and Smith DE (2000). System dynamics modeling applied to software outsourcing decision support.Softw Proc Improv Pract5(2–3): 169–182. Rus I, Neu H and Munch J (2003). A systematic methodology
for developing discrete event simulation models of software development processes.Proceedings of the International Work-shop on Software Process Simulation and Modelling (ProSim 2003), Portland, Oregon, USA, 3–4 May, 2003.
Scacchi W (2000). Understanding software process redesign using modeling, analysis and simulation. Softw Proc Improv Pract
5(2–3): 183–195.
Shannon RE (1975). Systems Simulation: The Art and Science.
Prentice-Hall: NJ.
Spiegel M, Reynolds PF and Brogan DC (2006). Grand challenge case studies in a simulation curriculum. In: Perrone LF, Wieland FP, Liu J, Lawson BG, Nicol DM and Fujimoto RM (eds).
Proceedings Winter Simulation Conference, Monterey, CA, Piscataway, NJ: IEEE, pp 2242–2249.
Standridge CR, Centeno MA and Johansson B (2005). Introducing simulation across the disciplines. In: Kuh ME, Steiger NM,
Armstrong FB and Joines JA (eds). Proceedings Winter
Simulation Conference, Orlando, FL, Piscataway, NJ: IEEE, pp 2274–2279.
Taylor SJE and Robinson S (2006). So where to next? A survey of the future for discrete-event simulation.J Simul1(1): 1–6. Terzi S and Cavalieri S (2004). Simulation in the supply chain
context: A survey.Comput Ind53(1): 3–16.
Waisel LB, Wallace WA and Willemain TR (1999). Visualizing modelling heuristics: An exploratory study. In: De P and De Gross JI (eds),Association for Information Systems.
Willemain TR (1994). Insights on modelling from a Dozen experts.
Opns Res42(2): 213–222.
Received 1 February 2007; accepted 19 October 2007
