AOAB Evaluation Framework Process

The AOAB Evaluation Framework (AEF) is a multi-step method to define each part of the framework using sets of criteria. Each criterion set defines the characteristics and main ele-ments of the AOAB. AEF requires improvement to remove inconsistencies, conflicts, overlap and the addition of criteria that cover more than one aspect. In the case of a particularly complex system or game “ making a choice from the apparently very wide range of methods and tools available can itself be a complex and costly process” [90] it is necessary to define a systematic evaluation framework in order to identify a suitable evaluation method. According to [143], there are three main methods that used to evaluate new techniques, methodolo-gies and tools, which are: surveys, case studies and experiments. A short definition and description of each of these is provided in Figure 6.1.

Figure 6.1: Evaluation Methods [143].

104

From [28], the Figure 6.2 describes a comparison between the evaluation methods employed in game engineering. The domains of the comparison are selected as follows: (a) Game software engineering is a subcategory of conventional software engineering. (b) traditional software engineering is a mature scientific area. (c) Agile software development is selected because it is a young domain and consequently complements traditional software engineering, with respect to area maturity.

The results of the study suggest that case studies are most frequently employed in Agile software development research. In addition, surveys are most frequently employed in game development research and finally, experiments are more frequently conducted in software engineering research. The AEF procedure will include surveys, case studies and experiments, as shown in Figure 6.3.

Figure 6.2: Comparison Between Evaluation Methods [28].

Most popular software engineering methodologies are created to be generic and suitable for many kinds of projects. The evaluation methods for the AEF are divided into two types:

qualitative and quantitative evaluation.

Figure 6.3: AOAB Evaluation Process

6.4.1 Survey Approach

This approach does not involve the practical use of evaluation. The survey can be used in many ways; sometimes, an organization or individual with experience is asked to use the methodology and then provide information about the methodology. This information can then be analysed using standard statistical techniques [82].

This technique is used in the present AOAB evaluation by conducting workshops in the academic and industrial sectors. This part of the research will be described in detail in the chapter Eight and Nine. Another form of evaluation survey is to compare the new methodology with similar existing methodologies. Methodology evaluation is a complex issue that is subject to various different points of view. The results of AOAB have already been compared in terms of development time, project management and the benefit of adapting two methodologies to create new hybrid methodology. In the evaluation methodology process, the following methodologies are selected to be compared with AOAB methodology:

1. DeLoach et al. [50] present an enhanced version of MaSE, called an Organisation-based Multi Agent Software Engineering (O-MaSE) methodology, to address a lack of indus-trial strength methods and tools to support multi agent development. Management and deployment issues are initially covered in O-MaSE, but not in a way that involves the customer in the process or accepts any change in development time, as used with

106

AOAB. Furthermore, the testing and evaluation phase is not included in the O-MaSE life cycle. The O-MaSE methodology framework is based on two meta-models: SPEM 2.0 and the O-MaSE meta-model. The SPEM meta-model defines methodology-related concepts, while the O-MaSE meta-model describes product related concepts [50].

2. Chella et al [41] present a hybrid methodology named PASSIAgile, to be used in robotics, the main idea of which is similar to AOAB, a hybrid between AOSE and Agile methodologies. Chella uses PASSI for the named code generation phase. This phase is largely supported by (Agent Factory) to automatically compile agent structure, patterns reuse and code generation [40]. For AOAB, the reason for selecting MaSE is its ability to define the agent goal in the initial phase. Furthermore, MaSE is able to add a new goal to an agent in any phase.

3. Jamont et al.[72] present a DIAMOND multi agent methodology that focuses on the hardware and software requirements of the system. The implementation phase consists of partitioning the system hardware and software parts to produce the code and the hardware synthesis. On the other hand, AOAB focuses only on the software require-ments part of game creation, although both use an iterative process.

According to [84], DESMET is a method for evaluating software engineering methods, and tools are mentioned the advantage and disadvantage of survey as follows:

Advantages of Survey

• They make use of existing experiences (i.e. existing data).

• They can confirm that an effect generalises to many projects/organisations.

• They make use of standard statistical analysis techniques.

• They require less time and effort than the formal experimental approach.

Disadvantages of Survey

• They rely on different projects/organizations keeping comparable data.

• They only confirm association, not causality.

• They can be biased due to differences between those who respond and those who do not respond.

• The difficulty of finding the right people to ask to participate in the survey, particular if the evaluated methodologies are not popular

Several tasks are associated with this survey, such as choosing the type of survey, for example a web-based survey or personal interview, building the survey documentation, such as a questionnaire, and identifying people to participate in the survey. Finally, the evaluators will run the survey and collect and analyze the responses according to the survey design [46].

In Chapters Eight and Nine, the workshop process will be covered in detail, from building the questionnaire, inviting participants, collecting data and, finally, analyzing the participant feedback.

6.4.2 Formal Experiment Approach

A formal real world experiment involves asking real companies to perform a task or play a game using the new methodology. The results will be analyzed using standard statistical techniques. For the AOAB methodology, a formal experiment has been conducted with undergraduate students completing a game design and implementation assignment. This part of the work will be covered in detail in Chapters Eight and Nine. Formal experiments are appropriate for exploring relationships; this approach is likely to produce the most reliable results, as it seems to reduce the influence of single assessor differences. It is, however, the most costly and time consuming approach [46].

108

6.4.3 Case Study Approach

The case study is similar to an experiment, but the level of control is lower in the sense that they are mostly observation-based studies [143][83]. Case studies is easier for a game developer organization to perform as there is no replication; the only limitation relates to the confidence that case study will allow an assessment of the true effect of the methodology. For AOAB, a case study approach is used to create a test drive game. The industry case study required approximately three months of work with an interested company who collaborated on the research free of charge as they are interested in improving their work and knowledge.

The number of participants involved in the case study was four individuals, including the researcher. This part of the evaluation will be covered in detail in Chapters Seven and Nine, in the critical analysis of the research results.

According to [84], DESMET is an appropriate method for evaluating software engineering methods and tools are mentioned the advantage and disadvantage of case study as follows:

Advantages of Case Studies

• They can be incorporated into normal software development activities.

• If they are performed on real projects, they are already “scaled-up” to life size.

• Provide a practical evaluation performed by an actual user of the methodology.

• They allow the researcher to determine whether or not expected effects apply in specific organizational and cultural circumstances.

Disadvantages of Case Studies

• With little or no replication, they may yield inaccurate results.

• There is no guarantee that similar results will be found in other projects.

• There are few agreed standards or procedures for undertaking case studies. Different disciplines have different approaches and often use the term to mean different things.