Objectives and Reasons for this Thesis

2.6.1 Need for Experimentation Support

In chapter 1. a concern was raised about the problems of experimental design and analysis in simulation. Throughout this chapter, the objective has been to identify what support is in fact available and failing that, in what context support can be provided. The theory behind simulation studies was investigated and a look was taken at how support was provided in the decision- problems of simulation. The following observations may be made.

Way back in 1959. Conway & al were stressing that "Perhaps the first lesson the neophyte simulator must learn is that research entirely by experimental methods is a painfully slow and difficult process even under the ideal conditions o f control which simulation provide". In 1968, Mize & Cox were still preaching that "Proper considerations o f statistical design methods can lead to significant reductions in computer time, and more importantly to improve interpretation o f data gathered from simulation experiments".

It is noted that, over twenty years later, there is still no proper support for the phase of experimental design and analysis. Grant (1986) "Simulations have to be interpreted by skilled OR scientists before a naive user can readily understand them".

Eventually, non-academics as personalised by Smith (1986) express their alarm at the dangers of misuse of simulation and the lack of support. Academics are not all surprised. Zeigler & De Wael (1986) "Most simulation practitioners have not had such exposure and consequently the results o f simulation studies are frequently suspect fro m a methodological point o f view".

Solomon (1982) related her experience : "Some senior people in the field believe that statistical analysis does not prove much, but demands a great deal o f effort".

The popularisation of simulation prompts Mathewson (1985): "As more users experience the ease with which models can be built, / believe that we shall need to strengthen training in the proper use o f simulation". This is justified by Moser (1986) who writes "Simulation does not provide solutions, it merely shows values o f a set o f variables over a period o f time given certain

assumptions. Interpreting theses values and drawing inferences from them lies beyond the scope and intent o f simulation. Nevertheless, interpretation and inferences are o f critical importance to the user. Furthermore, the interpretation and inferences o f simulation output is often complex and remains the exclusive domain o f experts in many cases". But even experts require assistance as Luker (1986a) notes "An expert needs more support than a language can provide; there must be an integrated software environment to support all stages o f the simulation process from requirements specification to analysis o f the results".

Support for the phase of experimental design and analysis is still in proposal form as sug­ gested by O’Keefe (1986b) "Given a domain expert with little knowledge o f simulation, a useful advisory system fo r the experimental design phase might give advice on the statistics o f conduct­ ing experiments".

2 .6 J Limitations of Current Research

As was discussed in the previous section, some authors’ have addressed this problem. Most significantly O’Keefe, Luker, Haddock and Reddy & al.

Luker and Haddock have presented support in the context of program generators. They have built a number of experimental frames that support experimentation in its execution. How­ ever, the user is still entirely responsible for the input and selection of appropriate frames. There is no support in this respect. The higher level strategies to address specific problems through experimentation is not supported either.

O ’Keefe only addressed a small example problem of experimental support His system was very limited in scope. It did not offer any support in the actual execution of experiments.

The work of Reddy & al offers more insight towards higher level strategies o f experimenta­ tion. Their work appears strictly limited to KBS models, since instruments, goals and constraints have to be constructed manually for each new application. This application is still geared towards the expert user in its current implementation.

2 .6 3 Proposed Research

There does not appear to be any doubt about the importance of support for experimental design and analysis. Many authors propose such support.

Balmer & Paul (1986). in their CASM project, talk about "statistical design and analysis issues which seem to be specific to systems simulation, including the proper employment o f vari­ ance reduction techniques, the determination o f appropriate run-lengths and the avoidance of bias due to transients".

Reddy (1987) on knowledge-based simulation suggest that the ideal KBS should among other features be able to accept a goal in the form of expectation and :

- select model;

- determine performance metrics;

- generate search space o f plausible scenarios;

- execute simulation model by controlled selection of scenarios;

- recommend a scenario that satisfies the stated goal.

Moser (1986) proposes the creation of an integrated decision-support system developed by combining the ability of expert systems, which allows them to analyse the simulation output and draw the necessary inferences.

According to Oren (1986a) in "Knowledge-bases fo r an advanced simulation environment",

an experimentation advisor has several functions:

1. specification of experimentation (including specification of variables to be observed, instrumented and monitored.

2. link a model to different experimental conditions.

3. sensitivity analysis

Whilst talking about directions to explore in artificial intelligence. Oren & Zeigler (1987) suggest knowledge-based intelligent systems which have an ability to perform simulation studies that can define several scenarios within which they can simulate a system to increase their knowledge

about:

- behaviour of the model;

- sensitivity of the behaviour of a model to parameters or operating conditions.

Lehman & a1 (1986) also considers a system in knowledge base modelling where the results of experimental applications are statistically and graphically analysed in a dialog with the user. This system however is still very much in proposal form.

2.6.4 Objective of this Thesis

The objective is to research into the development of a support environment for experimental design and analysis by considering the use of artificial intelligence techniques such as production rules, heuristics and expert systems.

From Shannon's (1986) analysis of future simulation it appears that there is no certain out­ come about the future of simulation modeling. However, there will clearly remain a need for support to experimental design and analysis. In due course, this may well be integrated into the environment where the model is developed. For now. it is proposed to build a support environ­ ment that accepts Zeigler’s contention (1976) that the model is to be separated from the environ­ ment where it is run. Such an environment would permit transportability from one simulation implementation to another.

Experimental design and analysis will remain a difficult task. It is not the objective of this thesis to resolve the existing problems by developing new expertise, but only to consider a frame­ work whereby the existing expertise may be made more accessible to support the simulation user. It is considered that this expertise may go beyond the application of simulation techniques and reflect expert behaviour in the experimentation process.

Further interest could then be satisfied such as considered by Phelps (1986) "Of imerest are ES that will carry out statistical work normally done by statistical specialists". Also, it could be considered capabilities of a system as described by Martins (1983) " The simulator should be able

to make inferences from observed events to some unobserved effects".

Knowledge could be shared by making it available in the form proposed by Davis and Lenat (1982) by allowing why, how, explain questions. Alternatively further knowledge could be gath­ ered from experts about the tactical issues of simulation.

The implementation would be in Prolog as there appeared from the literature much encouragement about its potential in addressing a variety of simulation issues using A1 tech­ niques . Additionally. Adelsberger believes that "the simulation expertise (running and designing experiments) can be performed in Prolog automatically". Shells will not be used since they are often found inflexible (d'Agapeyeff. 1984; Blakemore & al,1986) and this would constrain the research horizon.