Data collection is the key that enables the research to accomplish the correct evaluations more effectively. Data collection is advantageous as it provides the researcher with the chance to work with empirical data. Good data need to have three major criteria: they have to be reliable, relevant and representative. Reliability means that the data provide accurate and consistent calculations. Relevance means that the data capture the intended measurements. Representativeness means the data portray the service being measured. There are two well-known methods for collecting data: the quantitative and qualitative data collection methods. Many studies have reported on the differences between these two methods (Belk, 2006; Creswell, 2003; Jackson, 2009; Marvasti, 2004; Saunders, 2009).
Qualitative research is an in-depth study of social and cultural phenomena and an analysis of the quality of the human experience and, at the same time, it consists of non-numeric data or data that have not been quantified (Saunders, 2009). Qualitative researchers typically make passive observations with no intent of manipulating a causal variable (Jackson, 2009). It is a method of collecting data and tends to describe the meaning in-depth.
Quantitative research consists of the use of certain techniques that characterise the human experience in numeric data (statistics). It usually starts with a hypothesis for testing, observes and collects data, statistically analyses the data and, finally, draws conclusions (Jackson, 2009). Quantitative methods (e.g. questionnaires and experiments) deliver data that are easy to investigate statistically, and are properly consistent and reliable. The quantitative method is used when researchers need to make comparisons between two or more things and, at the same time, using numbers to show data.
There are many sources from which qualitative data can be collected. Generally, the most common source is the interview (which is often recorded using audio recorders and then the responses transcribed before an analysis is undertaken). Another data sources is observation and this method records the reaction of the participant (Cooper, 2012). Sometimes qualitative data collection can occur within surveys or case studies and within document analyses.
Quantitative data sources include experiments (this aspect is explored in detail in the next sections) and non-experimental studies and surveys (which use questionnaires and sometimes interviews for collecting data which is then analysed thereafter). Both methods (i.e. qualitative and quantitative) can be combined to collect data. One of the most common methods for collecting qualitative data are face-to face interviews. This method will be used during this research. At the same time, collecting quantitative data will not be neglected and, as an example, the researcher will record numeric data (quantitative data), such as the ratings by users in response to the usability questionnaires and the average time taken for completing search tasks.
Since this study compares two e-learning websites, one in Arabic and the other in English, with the aim of testing the level of usability of both versions it is crucial to describe the methods that will be used to investigate the web design features and in testing the usability. Investigating the user interface design can be achieved by using two methods: usability inspection methods that are performed by usability professionals and experts to investigate the user interface, and empirical methods that involve representative end users to evaluate the user interface.
3.11.1 Usability Inspection Methods
Nielson and Mack (1994, p.170) defined usability inspection methods as follows: usability methods“is the generic name for a set of methods based on having evaluators inspect or examine usability-related aspects of a user interface”. Usability inspection methods are one of the significant approaches in accomplishing usability tests. As was mentioned previously it is normal practice for an expert or a group of inspectors to try to detect usability problems in a website in order to improve the quality of the usability. Many approaches can be used to evaluate the usability of websites including cognitive walk-through, pluralistic walk-through, formal inspections and heuristic evaluation (heuristic evaluation is described in chapter two).
3.11.2 Empirical Usability Methods
In order to inspect and improve the usability of interactive systems, there are various empirical usability methods, which can be applied. The widespread methods for collecting data and user interaction experiences are observation, performance measurement, thinking aloud, and questionnaires. The observation method focuses on watching the user and his/her interactive actions. In essence, this method does not require a technology tool. More details on this method are given below.
3.11.3 The Observation Method
The observation method is used to find out what a user does, his/her work context and how technology assists him/her (Rogers et al., 2011). Sanders (2009) listed three types of observations: primary observation which focuses on observing what happens directly and recording these actions in diaries; secondary observation which focuses on the verbal statements of the user on what has happened, and experimental observation which focuses on user perceptions and impressions about user experience and recording
these perceptions in diaries. This study relies on empirical usability methods whereby users will be observed directly performing a number of search tasks using the e-learning website. The data collected from this study will include direct interaction with the system, user actions performed, user impressions, and feelings about the system following the study.
3.11.4 Performance Measurement
Martinez (2012, p.39) defined performance measurement as “testers or software tools record usage data and obtaining statistics during the test”. Indeed performance
measurement is one of the famous empirical usability measures that evaluate user performance objectively in order to shape the design of the interface. In addition, it is also used to assess if the usability objectives are satisfied and it measures the time that users take to accomplish tasks and counts the number of errors that are made by users. Moreover, performance measurement is usually used to compare different interfaces (Nielsen, 1993). In this research, performance measurement will focus on:
Calculating the average time (in seconds) needed to complete each search task.
Calculating the average number of clicks per search task for each e-learning version.
3.11.5 Thinking Aloud Protocol
Someren et al. (1994, p.1) defined thinking aloud as: “a method which consists of asking people to think aloud while solving a problem and analysing the resulting verbal
protocols”. In other words, a protocol enables the participant to talk aloud about whatever they are thinking and to verbalise their thoughts during their participation throughout the experiment. Thinking aloud is a very useful and valuable usability method as it helps the researcher to understand how users view the interface and how
they proceed to use it. It also shows which factors can affect the system and at, the same time, identifies the problems that require immediate attention by the designer. Likewise, it is a beneficial method of avoiding misinformation and obtaining direct data about the solution processes (Someren et al., 1994).
3.11.6 Questionnaires
Questionnaires are an indirect method of collecting the opinions of users about a particular system. Questionnaires can also measure the satisfaction of users and their predilections (Holzinger, 2005). There will be more discussion on the questionnaire in the section on the data collection process. In summary, this research will use a combination of the methods discussed above. It will observe user interaction with the e- learning website, collect performance data, and will ask users to verbalise their thoughts and complete post-study questionnaires.