A number of experiments and surveys were conducted to collect users’ experience on the proposed research on m-learning security issues and their perceptions on the interventions. Computer Science students and academic staff were asked to work with the app as case studies. During the experiments, log files were stored and user activities were logged to track the users’ behaviour on the system. The log files and the evaluation feedback were used for evaluating the app. At the conclusion of the experiment, participants were asked to complete a questionnaire provided along with the app so as to give feedback on the system and their overall experience. In addition, qualitative feedback was collected from academic staff who are experts in computer security, as part of overall feedback from users who were interested in giving further
54
suggestions. Details on data collection methods and justifications are described in the relevant chapters.
3.3.1 Quantitative
In this research, quantitative methods are used as strategies for data collection, which are normally accomplished by statistical methods to analyse findings. Questionnaire is a common approach which is use for obtaining quantitative data in a planned setup from participants (Smith 2015) and it is used in this thesis. Three types of questionnaire formats that are used when developing a questionnaire are: (i) structured, (ii) semi-structured and (iii) unstructured. Most of the time, the choice of the format is normally based on many factors, among them is the sample size. Cohen
et al. (2013) indicated that the more the sample size of the participants, the more structured the questions may become and as the research sample size was quietly moderate in all data collections, semi-structured questionnaires was used as one of the instruments to collect data in order to obtain balanced unbiased responses from the participants.
The benefit of using questionnaires is that it gives respondents the opportunity to state their opinions freely without any fear as their responses were unmonitored. Another advantage is that questionnaires can reach a large number of respondents effectively within a short time, thus it increases the response rate significantly. However, a disadvantage of questionnaires is that misinterpretation of questions may lead to inappropriate or irrelevant answers. In order to avoid this kind of problem, the researcher was available during the questionnaire session to attend to any queries that the respondents may have had in relation to the questionnaires.
Lastly, questionnaires may also be helpful in survey that involves a large number of respondents because they are likely to be more cost effective than other means. Before the distribution of the questionnaires to the participants, pilot tests are done with a small group of colleagues and the opinions and suggestions given by them
55
were taken into consideration in making the final copy of the questionnaires. A pilot test was conducted for the second time with another small group of colleagues in other to ensure high reliability and understanding of the questions.
Two sets of quantitative methods were administered during the research study, the first for the preliminary survey in order to gather users’ experience on the security issues in m-learning, the second was presented as an evaluation of the mobile learning security enhancement app. While the questionnaires were designed based on user-centred methodology approach, a Likert scale and System Usability Scale were used in the development of survey questions, as response options for the closed questions. The designs of the questionnaires are discussed fully in related chapters of this thesis.
3.3.2 Qualitative
Along with the quantitate method, qualitative data collection through interviewing was employed. Some semi structured interviews were used to obtain responses from academic staff as indicated in relevant chapters. The rationale for using semi- structured interviews was that, they most of the time, provide a relaxed interview environment which enables positive interaction among the participants while allowing the researcher to collect rich quality data as well as preserving a semi- structured interview rule. Another reason is that semi-structured interviews provide a flexible and conducive environment for participants unlike formal interviews which have a reduced flexibility and may sometime change the interview process into a formal one in which participants may feel that they are being pressured for responses (Taylor et al., 2015). As with the quantitative data collection, pilot mock interviews were conducted with colleagues and their opinions and suggestions were taken into consideration in making the final interview questions. Three sets of interviews were administered during the research study, the first for the preliminary survey in order to gather opinion on the security issues in m-learning, the second was presented to
56
understand if the findings are adequate and the last quantitative survey was an evaluation of the m-learning security framework and enhancement app. Details on the interviews are discussed in relevant chapters of this thesis.
The data analysis method employed in this research follows a thematic analysis approach which is based on examining themes within the data collected (Bryman, 2012). Initially, all the primary data collected through interviews and questionnaires were transcribed from a tape recorder before coding. A Pattern coding is a way of grouping summaries into a smaller number of sets, themes, or constructs, and coding pattern may be characterised by (i) similarity (that is things happen the same way), (ii) difference (that is things happen in predictably different ways) and (iii) frequency (that is how often or seldom things happen) and basically a set of structured data is an outcome of coding.
Nvivo qualitative analysis software package was used for coding the content analysis. In Nvivo coding refers to coding with a word or short phrase from the actual language found in the qualitative data record. After the raw data or notes taken by the researcher is entered into the word processing component of the program, the package assists the researcher in content identifying terms, phrases, or themes that appear in the text document. The extraction involves keyword search within the raw data in the software and then counting how many times they appear and in what context. This makes it easier for the researcher to convert the coded findings to standard statistical analysis to determine frequencies and correlations with the data and to make necessary reports thereafter in each section of this thesis.
3.3.3 Case Study
This research also makes use of case studies to collect empirical data necessary to gather valuable feedback on the findings. While Cohen et al. (2011) termed case
57
study as a challenging method, Yin (2015) considered case study as an effective method of using different procedures to correlate an argument that is relevant to the way of investigating activities that happen in a specific context. In this research, the case study involved participants using the security enhancement application during the evaluation stage and observing the activities being carried out in the process. The aim of the case study is to enable the participants to contribute to the development of the m-leaning security enhancement app providing personal opinion on it. Through interviews and questionnaires on the case study, learners also provided responses based on their perspectives on how the application improves their security understanding of m-learning devices. It is important to evaluate users’ satisfaction when using applications, especially in relation to their needs, therefore the case study, forms part of the user-centred evaluation methodological approach in the thesis.
3.3.4 Mixed method and triangulation
The user-centred methodology used in this research involves many field work activities carried out in university environments. Therefore, different data collection strategies were used to gather data and using more than one research method for data collection to achieve the research objectives is known as a Mixed Method. The mixed method of data collection used in this thesis employs both qualitative and quantitative methods described above as they are regarded as highly complementing rather than mutually exclusive to one another (Creswell and Clark, 2011). The mixed method of data collection allows the researcher to engage in a “triangulation approach” which involves using different methods of data collection, varying data sources, different analyses or theories to check the accuracy and validity of the findings (Lesser, 2016). The benefit of triangulation is that data obtained from multiple methods reduces the effects of limitations any one particular method may have on the data. The mixed method strategy research technique according to Bryman (2012), though it may be time consuming, it is chosen in this research to provide a balanced view of the research outcome.
58