SELECTING PANEL EXPERTS

The skills and experience of experts have a great effect on the validation results (Hvannberg et al., 2007); the more experienced a panellist is, the more pertinent the

163

evaluation outcome is. Requests were sent to many SONATRACH experts who have good knowledge in maintenance area but only twenty five agreed to participate. These experts were nominated to form a "Panel of Experts" based on their expertise in the field of maintenance at national petroleum company SONATRACH as well as number of working years. This panel consisted of three types of experts: three heads of maintenance department, eight procurement engineers and eleven maintenance operators. These professionals were approached firstly at their place of work and secondly through emails. Besides, three lecturers at the Algerian Petroleum Institute considered as academia experts were interviewed on the usefulness and applicability of the model. A written questionnaire (as shown in Appendix A) was used to obtain the experts’ judgement. Initial interviews were conducted with each of the panellists to prepare and guide them for the accomplishment of the validation questionnaire.

9.4.3 METHODOLOGY

Validation and test results about the usability of the model were collected on two approaches. First, the panellists ranked the model usability by means of a questionnaire. Then the same panellists were to test the model for qualitative analysis by collecting their comments and suggestions. Besides, an analysis of variance was carried out to detect any possible differences in expert assessment. The surveys study was structured around six phases:

1. Definition of the underlying theory and the model structure 2. Definition of the pilot study

3. Expert panel initialisation and discussions 4. Questionnaire building

5. Validation construction process 6. Validation result processing.

During the third phase, the initialisation and discussions were carried out with a reviewing panel of SONATRACH to come up with initial success criteria for model implementation. Firstly, emails were sent to the panellists to invite them to enumerate the measurement criteria that should be considered when validating models related to their field of expertise. Table 9.3 presents the chosen criteria and their frequencies given by the 25 respondents. In this review, usability was most frequently measured, followed by model output and model adaptation.

164

Table 9.3 : the measurement criterion of the model validation.

N° Measurement criterion Frequency over

25

1 Usability 17

2 Model output relevance 11

3 Adaptation 11

4 Simplicity 11

5 User expertise 10

6 User satisfaction 8

7 Input data 8

8 Model interface environment 7

Secondly, another set of criteria was provided to the experts from literature in system usability studies (Nielson, 1993, ISO 9241-11, 1998, Demers et al., 1996) and Shackel, 1991). The above criteria were completed and modified based on expert feedback to reflect the characteristics of maintenance environment. As a result, twenty six questionnaire items were generated in relation to the model validation and testing. These items can be classified under the following groups:

ƒ Variables concerning the model usability: usability is considered by ISO 9241- 11 (1998) as the degree to which a system can be used in a specified context to attain particular objectives with efficiency, effectiveness and satisfaction of use. Based on this definition, usability is therefore measured by means of three variables, namely effectiveness, efficiency, and satisfaction. In this validation study, these variables are defined as follow:

ƒ Efficiency: means the model capacity to generate satisfactory results with a minimum amount of required input data;

ƒ Effectiveness is the degree to which the model fulfils its intended goals or functions; and

ƒ Satisfaction reveals level of approval toward using the model. ƒ Variables concerning the model

165 ƒ Model structure

ƒ Model content ƒ Simplicity

ƒ Learnability and ease of use

ƒ Helpfulness and problem solving capabilities ƒ Variables concerning the model adaptability

ƒ User background and experience ƒ Familiarity with the theory ƒ Required input data

ƒ Usefulness of output data ƒ Missing parameters

ƒ Applicability at different whole life phases ƒ Adaptability to the organisation environment ƒ Adaptability to user culture

ƒ Model weaknesses and improvement

The questionnaire was reviewed by academia experts at the Algerian petroleum institute on a variety of aspects including technical, language and item redundancy. The questionnaire measure consists of 26 items clustered into 4 groups, namely model usability (3 items), model (6 items), model adaptability (8 items) and model weakness and improvement (4 items). The respondents graded these items using a 5 Likert-type scale from 1 to 5 where ‘1’ is the lowest and most negative judgement on the scale, ‘3’ is the average judgement, and ‘5’ is the highest and most positive judgement. The selected items are presented in Table 9.4. Fowler (2002) asserted that a Likert scale has the advantage to be easily understood and it well discriminates among respondents views. In addition, it requires short questionnaire items of a few lines. Finally, it is straightforward to analyse and interpret responses and the capability to get summated values.

166 Table 9.4 : identified success criteria.

Success

Criterion Purpose Question sample

Usefulness

Effectiveness Efficiency Satisfaction

I am successful in general in finding required data when using the model.

Overall, the model is useful in helping me

I achieve what I want using the model

Can the results obtained by the model be applied?

Adaptability to environments

Satisfaction with the adaptability features of the

model to environments How satisfied are you with the adaptability _{features of this model to environments?}

Adaptability to culture of users

Satisfaction with the adaptability features of the model to the users.

How satisfied are you with the adaptability features of this model?