Methods  of  Analysis

This section discusses the methods used for analysing both the qualitative and quantitative data collected from all deployments. Issues regarding the reliability and validity of these methods are also discussed.

6.5.1 Qualitative data

The data was generated from interviews, observations, and analysis of students’ submitted work as shown in Figure 25. The data from the interviews was analysed via assigning codes and themes as described by Miles et al.

(2014). It was decided that the Content Data Analysis method would be used to allow patterns to emerge from the interview data. This process included grouping together the responses from participants for each interview question to enable themes to emerge from the grouped responses. The themes were then given appropriate names that related to the discussed issue. This

qualitative data analysis method is commonly used for evaluating interview transcripts (Cohen et al., 2007). Narrative description (Miles et al., 2014) was used to present the results.

In-depth content analysis of coursework submitted by students was used to discover what issues students tended to discover while conducting their observations; the frequency with which particular issues occurred in the work was also noted. The Descriptive Coding method (Miles et al., 2014) was used to summarize issues identified in the students’ work. This analysis was conducted in addition to the application of the marking scheme used to allocate marks given by the lecturers to enable a more detailed analysis of the work.

6.5.2 Quantitative data

Quantitative data was generated from the questionnaires given to students from the HCI, UX, DUE, and Engineering modules. The purpose of these questionnaires was to evaluate the usability of the app interface and pedagogical usability of the sLearn app from the students’ point of view. The questionnaire for each deployment is explained in the evaluation design section of each deployment (see sections 6.1.2, 6.2.1, 6.3.1, and 6.4.3). The data gathered from the questionnaire included, students’ self-diagnosis of their level of Android expertise, SUS results to measure the usability of sLearn, and pedagogical usability statements. The pedagogical usability questions were different for each deployment as they related directly to the coursework assignment and specific changes in the design of the sLearn app resulting from the in-context evaluations. The SUS part of the questionnaire

was analysed using the specific formula that generates a usability score for each participant (Brooke, 1996). The average for each pedagogical usability statement was calculated to show the mean score. Additional analysis was conducted to understand the effect of the level of expertise of using Androids on pedagogical usability. To ensure a large sample size for the analysis, the HCI and UX questionnaire responses were merged, bringing the total number of respondents to 38. A cross-tab using a Chi-square test was performed to find out if there were statistically significant differences in responses between students who were “expert” and “non-expert” Android users.

6.5.3 Research Validity

To ensure the ecological validity of the results, it was crucial that the research methods aided in answering the research questions.

As the contextual blended learning model for the HCI module was new, it was not possible to use a previous cohort’s results for comparison. Also, it was not possible to easily create control groups for the assessed work in the cohort, as it could create an unfair advantage for the student who had the assistance of the app. The approach adopted was therefore to compare performance of work supported by the app, with other elements of the assignment work completed without the support of the app. The use of Vavoula and Sharples’s framework (see sections 3.3, 6.1.2, and 6.3.1) helped to distinguish benefits of the app from different perspectives. The use of established metrics for usability and pedagogic utility strengthened the reliability of the evaluations considering the effectiveness of the app design and perceived benefits.

The findings were further verified by triangulating results from the analysis of Groups’ presentations, questionnaire, and lecturers’ discussion and feedback to students.

It should be pointed out that there was an attempt to gather more data regarding HCI students’ perspectives and experiences of using the app, and develop a deeper understanding of reasons why some groups chose not to use it, via a focus group. However, due to a lack of engagement of the students to participate in a focus group, this could not be organised.

However as part of the DUE evaluation study, a more in-depth understanding of the use of sLearn app was possible. All evaluation aspects were considered to ensure that a maximum number of issues influencing use and experience were discovered. Choosing participants that had similar profiles to the actual end-users was necessary for an assessment of the app as whole.

Observing the participants interact with the app in the intended environments and allowing them to communicate their feelings via the ‘Think Aloud’ method provided insights into the user experience. Additionally, conducting an interview to follow up the observations was a very important part of this evaluation study. According to Taylor et al. (2002): “Interviews can provide rich data and give considerable insight into perceptions and attitudes.

Misperceptions or misunderstandings about what is being asked can be recognised and dealt with at the time. The interviewee has the opportunity to express opinions important to them, clarify ideas and feel that these are valued. The interview can be a learning process for both interviewer and interviewee”. Adding the SUS questionnaire aided in understanding the

participants’ views on the usability of the app. Furthermore, triangulating the results of these methods helped maximise the validity of the study.

The UX and Engineering deployments were conducted to further understand the effectiveness of sLearn as described in 6.3.1 and 6.4. However, the way the lecturers designed the assignment influenced the evaluation design and results as discussed later in 7.4, 7.4.5, and 7.5.

6.6 Conclusion

This chapter explained the research methodologies used to answer the main research questions and how the sLearn framework was tested with three different student cohorts: HCI, UX, and Engineering. The main studies were the HCI and UX, while the Engineering study was conducted to address the generalizability of the contextual blended learning model. Another crucial study was conducted to understand the effects of the environment on the user experience of sLearn. This was conducted with Masters level students enrolled on the DUE module. For an app such as this, conducting in-situ evaluations is vital. It is not only the user’s interaction with the app interface that is important to evaluate, but also the effect of the environment surrounding him/her on the usability of the app and how it affects his/her ability to observe and analyse. The next chapter will discuss the results and analysis of each of the studies.