It is crucial before starting to collect data to answer several important questions in order to identify the required data and the sources of these data, for example what type of data to collect, which kind of data collection methods are appropriate for this thesis, who are the stakeholders in this research? Answers to these questions are the key to collecting the relevant and appropriate data. This thesis will rely on three different data collection techniques: questionnaire, interview, and observation.
3.13.1 Questionnaires
A questionnaire can be laid out in a variety of ways and it can be used in many diverse conditions. A questionnaire is a range of questions that is used to elicit answers from users to help investigate research questions. The questions are usually divided into unstructured or structured questions. Unstructured questions ask the participant to provide a response in the way that they feel most comfortable, whereas structured questions ask the participant to select an answer from a given set of choices (Bhattacherjee, 2012). Delivering the questions in a similar way to a variety of people is an important step in most survey research. Questionnaires are divided into two types: self-administered questionnaires and interviewer-mediated questionnaires.
Questionnaires differ in the way they are developed and in how users are involved to complete them (Saunders, 2009). Self-administered questionnaires are completed by users whereas interviewer-administered questionnaires are completed by an interviewer using the users’ responses. Self-administered questionnaires are divided into four categories: Intranet-mediated questionnaires; these are developed and completed electronically.
Postal or mail questionnaires: these are posted to users who complete them on paper and return them to the interviewer.
Delivery and collection questionnaires: these are handed to users who answer them in their own time and they are collected once completed.
Other types of questionnaires include telephone questionnaires which are undertaken by telephone. This type of questionnaire is not relevant for the purpose of this research as this study will be carried out in a usability laboratory
where participants’ interaction with the system can be observed and recorded using appropriate software applications (e.g. Camtasia).
The empirical study in this research uses self-administered questionnaires which are handed to the participants at the start of the experiment. The participant is then asked to complete the questionnaire and return it at the end of the experiment. Additionally, this research uses an interviewer-administered questionnaire at the end of the experiment. This questionnaire is performed face-to-face to discuss any issues encountered during the interaction with the e-learning website and to explore any particular impressions and reactions to the systems. The questions in this questionnaire will be based on previous similar studies in this domain (e-learning, usability, web design and aesthetics) and on the literature review in chapter two of this thesis. The questions in questionnaires should be clear, should use simple language to avoid any misunderstanding and should be relevant to the e-learning websites. The following section describes the questionnaires that will be used to elicit user satisfaction and subjective ratings on the two versions of the e-learning website.
This thesis uses the following user questionnaires to assess e-learning outcomes, the usability of the system, aesthetics and user performance as described below.
Evaluation Questionnaire: this questionnaire (see Appendix A) is concerned with
five parts (general questions, colour, language, font and culture). More details on this questionnaire are provided in the next chapter. Overall, this questionnaire has twenty five questions with three primary objectives:
To gain a better understanding of the demographics and the characteristics of the Arab users (e.g. age, gender, etc.) and how these influence the e-learning website.
To investigate the influence of colour, font and other elements, generally, as used in the e-learning websites on user judgment.
To understand the role of language and culture on the reactions and impressions of Arab users towards some specific symbols, pictures and colours. This understanding will help create usability guidelines that are appropriate for designers aiming to develop e-learning websites for Arab users. These guidelines will empower web designers in understanding the needs and characteristics of Arab users and how web design features influence their e- learning experiences.
E-learning Participant Questionnaire: this questionnaire (see Appendix B) was
originally designed by The University of Northampton research department. This questionnaire focuses on the reaction of users to the e-learning websites following their use of both versions of the system. It explores the key features that affect users’
judgments of the system. This questionnaire contains twelve questions that mostly concentrate on how they affect the users of e-learning websites, what kind of issues they encounter during the use of the e-learning websites, andon users’ suggestions on how to
improve their interactive experience with the system in the future.
Usability Questionnaire: this questionnaire (see Appendix C) contains fifteen
questions which focus on the usability of the system (e.g. how easy it is to use the system), the advantages and disadvantages of each version of the system in the way affect the reaction of the users (both Arabic and English versions). This questionnaire enables the development of a clear understanding of the key differences and similarities between the Arabic and English versions and how they affect the users of e-learning websites. This questionnaire (a Computer System Usability Questionnaire) is based on
the computer system usability questionnaire which was developed in 1993 by the IBM company which was published in 1995 (Lewis, 1993). Since then it has been widely used by researchers around the globe. However, the questionnaire will be slightly modified in order to be appropriate for the target audience by deleting some of the questions that are deemed to be inappropriate for this thesis.
Aesthetics Questionnaire: this questionnaire (see Appendix D) looks at the beauty
of the websites and how they influence the component characteristics of the website (Li & Yeh, 2010) and, at the same time, this questionnaire attempts to capture the extent of the aesthetic impact on Arab users. The questionnaire consists of fifteen questions based on Noam Tractinsky’s questionnaire (Tractinsky & Lavie, 2003) but with some basic changes made to relate it to the objectives of this thesis.
3.13.2 Interviews
Marczyk et al. (2005) argued that interviews are a form of self-reporting and are an efficient way of collecting a variety of data. It is required that all the participants are asked the same questions in the same order and in the same kind of timespan to evade any dissimilarity in the collection of data. Interviews can take different forms depending on the field of research. Interviews focus on studying and exploring the behaviour, opinions, feelings and actual experiences of the participants. Questions in interviews enable the interviewer to understand phenomena that are relevant to the research in an organised manner. The types of questions differ depending on the purpose of the
research. Questions can take the form of ‘how’ and ‘why’ to find out about participants’
3.13.3 Structured Interviews
A structured interview is based on pre-set questions that should be standardised and identical. These questions are asked of the participant and the researcher records the responses. Structured interviews are also called quantitative research interviews (Saunders, 2009) and are very useful to use in experiments in order to collect data from participants. In this type of interview, all participants are asked the same questions in the same order and using a common rating scale (U.S. Office of Personnel Management).
3.13.4 Semi-Structured Interviews
A semi-structured interview falls between unstructured and structured methods. In this type of interview, the researcher usually has a list of questions and asks the participant to respond to these specific open-ended questions (Zikmund et al., 2009). The questions are divided into sections. Semi-structured interviews offer flexibility and allow the interviewer to add or abandon specific questions during the interview depending on the progress of the interview. The data generated during the interview should be recorded by audio-recording the conversation (Saunders, 2009). The participant has ample freedom to respond to the questions and the researcher has a schedule in order to keep the participants on track with the topics and subjects that are of interest to the researcher. The researcher uses semi-structured interviews to offer flexibility to the participant (to allow greater flexibility in the responses) and thus gain the benefits of a face-to-face interview.
3.13.5 Unstructured Interviews
Unstructured interviews are an informal method that can be used to explore the area being investigated in-depth. They also provide a flexible format, which allows the
researcher to explore specific points that relate to the research objectives and questions. The questions in an unstructured interview emerge from answers of the participant to specific questions by the researcher picking up on specific cues and themes (Fisher, 2007).
In this thesis, researcher will use semi-structured interviews at the end of the empirical study to gain further insights intothe participants’perceptions of, and the judgments on, the two versions of the e-learning website. The questions will focus on points relevant to culture and language and how they influence the participants’ judgment of the web design features such as colour, font and pictures. The questions will also shed light on the overall e-learning experience in both versions. The interview will start on these points and then explore further themes and topics as they emerge from the participants. Moreover, the interview will address any interaction and usability problems observed during the tasks.
3.13.6 Observation Methods
Observation is a technique for collecting and gathering information on participant behaviour, interaction and events relating to a specific phenomena or process. There are three primary types of observation techniques:
Covert observation: this is when the participant does not know that the observer is observing him/her. The observer is hidden in such a case. This technique is also known as the unobtrusive observation technique. In this type of observation, participants are more likely to act naturally.
Overt observation: this is when the participant knows that the observer is observing him/her. The observer is known to the participant in this case. This is also known as the obtrusive technique.
Researcher participation: this is when the observer takes part in the actual experiment with the participant to obtain a greater understanding of the process being studied.
Observations have many advantages. In observations, data are collected when an activity or user interaction is happening and observation enables the viewing of the real behaviour of the actual users. On the hand, observations might be affected by the
observer’s bias and are known to be time consuming. In addition, participants who are
being watched could perform better because they know they are being watched.
It is judged that the overt observation technique is the most appropriate technique for this study as it allows the researcher to observe and take notes of interesting user actions and issues. These issues will be discussed and explored further in the section on the semi-structured interviews.
3.13.7 Think Aloud Technique
In this experiment, qualitative data and user opinions will be collected (face-to-face) using the think aloud protocol. This technique enables participants to verbalise their thoughts in respect to their mental planning, their processes, impressions and feelings about using the e-learning website. Of particular interest is how the participants undertake specific tasks, what kind of issues they encounter and how they judge the qualities of the two versions of the e-learning website. The data generated through the think aloud technique will be recorded for follow up analysis.