Pilot study

To carry out the evaluation and test the system on normal users, it was necessary to carry out a pilot study. This is essentially to check if the system is working well, and to receive any comments that could be implemented before carrying out the final experiment.

3.8.1 Purpose

The pilot study is a smaller model of the final study. It tests how to gather information prior to a larger study and helps in validating the questions and the environment of the study. It is carried out to ensure that anything that may go wrong can be fixed before starting the final study. The pilot study will improve the latter‘s quality and efficiency.

There are four goals for the pilot study:

2. To ensure that the questions are serving their purpose and determine their suitability and appropriateness, i.e. to test that each question measures what it is intended to measure.

3. To ensure that the system is categorising and dealing with the experiment correctly to prevent errors from happening in the main study.

4. To avoid wasting time and money on a system and survey that is inadequately designed, so that the main study will not have to be repeated.

3.8.2 Evaluation tools 3.8.2.1 Survey

The survey was used to collect both quantitative and qualitative data i.e. closed-ended questions and open-ended questions. The closed questions were suitable particularly to test and prove the hypotheses. Surveys were used as they are easier to distribute, easier to collect, and easier to analyse.

The survey is split into three sections:

1. Section one is dedicated to collecting information regarding the personal information of the participants (demographic data).

2. Section two is dedicated to collecting information regarding the learning styles. The researcher employed the Likert scale for this section: ―one of the most widely used technique to measure attitudes‖, according to Ary et al (2006) [208]. According to Bryman (2004) [207] , the Likert scale ―is a multiple- indicator or item measure of a set of attitudes relating to a particular area‖. The options available to answer a closed question ought to be balanced, for example, for question number one, a choice of three answers were given (I know it very well, I have heard about it and I have never heard about it). The answers to the remaining questions were given in the form of a five point Likert scale, ranging from 1 (strongly disagree) to 5 (strongly agree).

3. Section three is dedicated to collecting information regarding the usability of the system using open-ended questions. Schuman indicated that to get the participants‘ true feelings on a topic open-ended questions are more effective than closed-ended ones [210].

The survey is web-based and was produced as part of the system (see chapter 5 section 5.4 for the full questionnaire). Questions one and three were employed to examine if the user had previous experience in learning styles. These relate to section 2 described above. The answers to these questions will help identify if there is a difference between those who do or do not have prior knowledge to recognise the lesson that was presented according to their learning styles; that will show whether the system is working properly or not. Question two is the central question used to test the hypothesis and research question to study the feasibility of the system. Questions four and five were used to examine the usability and the ease of use. These relate to section 3 above. Open-ended questions were included to allow participants to provide more information in their own words.

The fundamental goal of this survey was to obtain a general impression of the web- based system that adapted to two models of learning styles Findings from this exploratory study and the interviews described below (section 3.9) were then used to derive the statements of the final survey.

3.8.3 Procedure

300 people ranging from high school pupils to undergraduate and post graduate students were emailed asking them to participate in the pilot study. 157 of them agreed to participate in the study. The data was gathered as follows:

Users registered onto the system

Demographic information was gathered Two learning styles tests were completed

The results of these tests were used to build a profile for the user A lesson was presented according to the users profile

Users completed the evaluation survey

3.9 Interviewing experts

Upon completing the pilot study, interviews with educationalists were conducted. Although the expected information from an interview is richer, there are two main

drawbacks to carrying out interviews; they are time consuming, and harder to conduct and analyse compared to a survey. Furthermore, they depend on the participants, if they are open minded, forthcoming the interview will be a discussion more than merely questions and answers. This is the main reason why interviews were only used for part of the study and with very few participants.

For the interviews, sampling was carried out with a purpose, as ―the researcher samples on the basis of wanting to interview people who are relevant to the researchers questions‖ [207]. An argument from Silverman (2001) [211] is that ―purposive sampling allows us to choose a case because it illustrates some features or process in which we are interested. It demands that we think critically about the parameters of the population we are interested in and choose our sample case carefully on this basis‖. In the qualitative section of this research, the purposive sampling approach is based on choosing samples that achieve the researcher‘s goal, resulting in "gaining deep understanding of some phenomenon experienced by a carefully selected group of people" [212].

3.9.1 Purpose

The interviews aimed to review and discuss results of the survey, pilot study, and provide the researcher with suggestions to assist the final study. Interviews with a small group were used to achieve two main goals:

1. To test, improve and enhance the final study following the pilot study. 2. To gain educationalists‘ opinions, comments, and suggestions.

Four experts were involved in the evaluation of the various steps. They were asked several questions, which addressed the following points:

1. What is your opinion about the appropriateness of the presentation for individual users according to their learning styles?

2. How efficient is this method in terms of clarity and ability to test hypotheses? 3. Do you have any additional comments and suggestions?

Interviews were used as they enabled the researcher to go into more detail with the interviewees. The interviews were designed as semi structured to offer more flexibility allowing the researcher to explore issues as they were raised. Questions in the interviews can be explained more and if there are any ambiguities they can be resolved more easily. Participants in interviews can be encouraged to ‗tell the whole story‘. This type of method offers the flexibility to gain as much information as possible from the interviewee who is expected to point out things not anticipated by the interviewer.

3.9.2 Procedure

After agreeing to participate, the survey was delivered to four Ministry of Education reviewers for their validation of the contents. One reviewer is a Director of Educational Development, and the other three are lecturers, two from the Department of Computer Science and one from the Department of Flexible Learning. As part of the interview, the interviewees were told how the experiment with users (participants) was to be conducted and the experimental steps that would be taken. The stages of the interview process are as follows:

1. Interviewees were emailed asking for permission to participate and to arrange an appointment, they were also given a sample of the questions

2. They were introduced to the selected learning styles

3. A demonstration of how the system presents tutorials was given 4. They were shown what learners did in usage

5. Before the conducting the interview, they were given a consent form to sign 6. They were given a short introduction about the research study, and the system 7. They were subsequently questioned:

Demographic information (age, gender, level of education) Opinion about the evaluation method.

Opinion about the results of the pilot study. Opinion about the main questionnaire. Suggestion for the final study.

It was requested that all of the reviewers answer various questions as suggested by [213]. The questions were adjusted and asked accordingly in relation to the current study. Questions asked covered the interviewees opinions on the system and the evaluation methods, drawbacks related to the system or experiment, improvements that

could be made, the success of the research method in addressing the research questions and any final suggestions they would like to add. Overall, the Ministry of Education specialists responded with the appraisal that the questionnaire achieved its measurement goals.