Chapter 3 The BRACElet Project
4.3 Research design for this study
4.3.3 Philosophical approach chosen as paradigm
4.3.5.3 Phase 3: Iterative cycles of testing and refinement of solutions in practice
Two cycles were used to test and refine the tutorial to teach students to draw variable diagrams. The first cycle is described in Chapter 6, and the second cycle in Chapter 7. Mixed methods were used in the data collection and analysis in these cycles. The qualitative method was used to explain, elaborate, illustrate and clarify findings generated by the quantitative results (Greene, Caracelli & Graham, 1989). This provided contextual understanding for the quantitative findings, and assisted with a more
comprehensive account of the inquiry. It also triangulated findings and thereby provided greater validity (Bryman, 2006).
Table 4.3 Phases of design-based research mapped against elements that need to be addressed in the research with the corresponding positions and methods in the thesis (adapted from Herrington et al., 2007:3)
Phase Element Position Method
Phase of design- based research (Reeves, 2006)
Topics/elements that need to be addressed
Position in this thesis Method used to implement element PHASE 1: Analysis of practical problems by researchers and practitioners in collaboration
Statement of problem Chapter 1, Section 1.3.1 Chapter 4, Section 4.3.2.1
Development of proposal
Consultation with
researchers and practitioners
Chapter 4, Section 4.3.2.1 and section 4.3.4.3
Research questions Chapter 1, Section 1.3.3 Chapter 4, Section 4.3.2.3
Literature review Chapters 2 and 3, Section 5.2 Literature review PHASE 2: Development of solutions informed by existing design principles and technological innovations
Theoretical framework Chapter 5, Section 5.2 Literature review to provide theoretical framework Development of draft
principles to guide the design of the intervention
Chapter 5, Section 5.3 Synthesizing framework of guidelines from theoretical framework
Description of proposed intervention
Chapter 5, Section 5.4 Design and develop PV tutorial PHASE 3: Iterative cycles of testing and refinement of solutions in practice Implementation of
intervention (First iteration) Participants
Data collection Data analysis
Chapter 6 First implementation of PV tutorial with evaluation using statistical analysis and usability study in HCI laboratory
Implementation of intervention (Second and further iterations) Participants Data collection Data analysis
Chapter 7 Second implementation of PV tutorial with evaluation using statistical analysis and usability study in HCI laboratory PHASE 4: Reflection to produce ‘design principles’ and enhance solution implementation
Design principles Chapters 5, 6 and 7 Reflection on literature to extract design principles and on analysis of
implementation to enhance PV tutorial
Designed artifact(s) Available on course website10
Professional development Chapter 8 Reflection on professional development
10 The tutorial is available on the following link:
http://osprey.unisa.ac.za/download/Disk/tuts/COS1512/Variable%20diagram%20application%20%28for%20do
An important ethical consideration in this research was that the study should not impact negatively on any student by preventing access to the tutorial, similar to the principles followed by the BRACElet project as discussed in section 3.4.1. Therefore, it was not possible to use a control group as such, and factor analysis was done instead to determine the effect of a number of factors, one of which was students’ use of the tutorial, on their final marks.
The two cycles used to test and refine the tutorial are presented as follows in this research report: • First cycle: The tutorial was first deployed in 2012 by incorporating it in the study material
for COS1511, for both the first and second semester of 2012. This constituted the first iteration of testing and refinement. Chapter 6 reports in detail on this cycle.
• Second cycle: The second iteration of the implementation occurred in 2013 and is discussed in Chapter 7. It followed a similar structure to the first cycle, after refinement of both the tutorial and data collection instruments based on the findings of the first cycle.
The characteristics of the participants, data collection and data analysis were similar in both cycles, and are briefly discussed below.
Participants
All students enrolled for COS1511 in 2012 and 2013 were participants of the research. Students registered for 2012 acted as participants in the first cycle and students enrolled in 2013 were participants in the second cycle. In the first semester of 2012, 2 877 students registered for COS1511 and in the second semester of 2012, 1 921 students. In the first semester of 2013, 1 623 students enrolled for COS1511 and in the second semester of 2013, 1 593 students11. During each of the two cycles, a number of these students were observed and eye tracked in the HCI usability laboratory at Unisa while using the tutorial to evaluate the user interface and find usability problems in the design (Nielsen, 1994). This was done to increase the effectiveness and utility of the designed intervention. Self-selection sampling was used, since participants had to come to the usability laboratory at the Muckleneuk campus of Unisa for the observation and eye tracking. Self-selection sampling occurs when researchers advertise their need for respondents on a particular topic and collect data from anyone who responds (Oates, 2006). It should be noted that these participants might not have been representative of the wider COS1511 population, since they might have thought it would be to their benefit to participate in the observation and eye tracking /usability testing of the tutorial (Oates, 2006).
11 These numbers differ from those in Table 1.1 in Chapter 1, since Table 1.1 reflects only students still registered for the module at the time of exam sitting, and does not include students who cancelled the module. It was not possible to determine whether students cancelled the module before or after they had completed the
Data collection
The data collection of the research consisted of the following:
• An optional multiple choice questionnaire (see Appendices B.1.1 and B.1.2) on the tutorial, to be completed with or after the first multiple choice assignment for COS1511 during the first cycle. A number of questions that require the use of variable diagrams were asked in this assignment, which should have encouraged students to use the tutorial to learn how to trace. The questionnaire contained questions on biographical data and previous experience, how students used the tutorial and their experience of using the tutorial. For the first semester in the second cycle two questionnaires were administered, one similar to the questionnaire used in the first cycle (Appendix B.2.1.1), to be completed with or after the first multiple choice assignment for COS1511, and another to be completed at the end of the semester (Appendix B.2.1.2). In the second semester of the second cycle only the second questionnaire from the first semester in the second cycle were administered (Appendix B.2.2.). In the questionnaires administered at the end of the semesters the questions from the first cycle were adapted or expanded to shed more light on the findings of the first cycle.
• The examination marks for each student.
• Direct live observation with eye tracking of a small number of COS1511 students in the
HCI
laboratory while using the tutorial. One participant at a time was observed. During observation, participants interacted with the tutorial and conducted specified learning tasks, including a retrospective comprehension test (see Appendix C.3 for a list of the learning tasks). Participants were observed in real-time to note their actions, menu selections, reactions and body language. Video recordings of the interaction were made for subsequent re- observation and in-depth analysis. Time taken on tasks, reflection, exercises, errors, and so on, was electronically recorded. The observations and eye-tracking (in particular, gaze replays) were used to determine how students use the tutorial.• A questionnaire with open-ended, semi-structured questions (Appendix C.4) that students completed after the eye tracking and observation. The same questionnaire was used in both the first and the second cycle.
The multiple choice survey on the tutorial as well as assignment and examination marks were collected for both semesters in 2012 and 2013, while observation and eye tracking of students in the usability laboratory were done once in 2012 and once in 2013.
In this study, questionnaires (Appendix B) were used to collect data about the students’ use and user experience of the tutorial, since there were a large number of wide-spread participants; and standardised data were required for the statistical analysis. The questionnaires were self-administered, with closed questions and pre-defined answers to make it easy to complete and analyse. The
researcher used both her study leader and a professional statistician as sounding boards to assist with ensuring the content and construct validity as well as the reliability of the questionnaire. Content validity refers to making sure that the questionnaire generate data about the concepts being investigated, while construct validity is concerned with whether the questions measure what they are expected to measure. Reliability is concerned with whether the questionnaire would generate the same results if repeatedly administered to the same students (Oates, 2006). Construct validity are determined by Exploratory Factor analysis on the constructs in a questionnaire tested by a Likert scale (Terre Blanche et al., 2006). In the first implementation cycle there was only one construct, namely perception of positive experience, termed ‘positive user experience’. Therefore construct validity could not be tested. In the second implementation cycle another construct was added to the questionnaires, namely perception of learning from the tutorial, termed ‘learning experience’. However, since there were not enough responses to allow for exploratory analysis, construct validity could not be tested. Reliability of both of these constructs, that is, the internal consistency (Terre Blanche et al., 2006), was however tested by item analysis. The disadvantages of using questionnaires in this scenario included not being able to correct misunderstandings, offer explanations or help, query disparities between answers or check the truthfulness of answers. The pre-defined answers might have frustrated participants so that they refused to partake, and could also have biased respondents to the researcher’s viewpoint (Oates, 2006).
While the questionnaires gave an indication of user experience, eye tracking in a usability laboratory was used to observe how students used the tutorial, and so investigate usability. Eye tracking consists of recording and measuring eye movements with a screen equipped with a built-in eye tracking device. Most eye tracking studies analyse patterns of visual attention of individuals when performing tasks such as reading, scanning or searching. Eye-tracking equipment allows for recording and replaying the records of eye movement, while software identify and analyse patterns of visual attention when performing a task (Djamasbi, 2014).
People only see when they focus overt attention on an object. A pause of eye movement on a specific area is called a fixation. The rapid movements between fixations are called saccades. Since information can only be perceived and interpreted information when the eyes fixate on it, there will be more or longer fixations on more complicated, confusing or interesting features. The data created by eye tracking consists of gaze plots and heatmap visualizations. A gaze plot shows the movement sequence and position of fixations and saccades on an observed image. The size of a dot gives an indication of the duration of a fixation, while the dots are numbered in the order in which fixations occur. A heatmap highlights the areas of the image where participants fixated. Warm colours such as red or orange will indicate longer fixations or many fixations on the same area (Djamasbi, 2014). Since eye-tracking data only provides an indication of how long and where a participant fixated, it does not give an indication of the success or comprehension of the participant. Additional
performance measures such as retrospective comprehension tests or think-aloud feedback from the participant is required to correctly interpret the data (Hyönä, 2010). In addition, the following had to be kept in mind to enhance validity in eye tracking (Poole & Ball, n.d.):
• Eye trackers can have difficulty tracking participants with glasses or contact lenses since that interrupts the normal path of reflection. Large pupils and lazy eyes may also be problematic. • Participants in eye-tracking should perform well-defined tasks to attribute eye movements to
actual cognitive processing.
• Visual distractions should be eliminated.
• Filtering and analysis should be performed automatically to process the huge amounts of data generated by eye tracking, en eliminate introducing errors with manual data processing. Pernice and Nielsen (2009) recommend using gaze replays with six participants for qualitative eye tracking. Gaze replays enable the researcher to watch the test session video with the user’s eye motion overlaid over a recording of the changing computer screen image. The user’s eye is represented by a blue dot moving around the screen. Gaze replays are the most accurate method for analysing the information since the replay can be slowed down to really see everything the user looked at in the order in which they were doing it. However, they are very time consuming since the eye moves so quickly that segments frequently have to be slowed down and replayed to get an accurate idea of what happened (Pernice & Nielsen, 2009).
Heat maps and gaze plots can only be used when all participants follow the same path through an application. Therefore only gaze replays were used to investigate how participants used the tutorial, as each student followed his or her own path through the tutorial. After eye tracking, the participants were asked to draw variable diagrams in a retrospective comprehension test to give an indication of the comprehension or learning of the participant.
Data analysis
The main objective with the multiple-choice questionnaire included with the first assignment, was to investigate the effect of the tutorial on examination marks. As far as the data analysis was concerned, statistical analysis was used to investigate several factors that may affect students’ performance, such as the biographical properties of students, their positive user experience and their use of tracing. The extent and manner of their tutorial use were investigated in relation to their examination marks. A trained statistician performed the analysis to ensure its validity. It should be noted that since the survey was adapted and expanded, more factors were investigated in the second cycle.
Statistical analysis has also been used to compare the examination results of the participants with the marks for questions on reading (tracing) program code, explaining program code and writing program code to determine whether the students’ ability to read code (that is explain the purpose of code) and
trace code had any effect on students’ results. This was done to confirm that the findings from the BRACElet project that was used as the departing point for this study, applied to the participants in this study.
The live observations in the usability laboratory and the questionnaires completed after the observations were used to provide further insight into the results of the statistical analysis. Tracking the eye movements of participants using the tutorial provided knowledge of how students interact with the tutorial. This identified usability problems with regard to the layout of the tutorial, terminology used, students’ comprehension of the material presented and their navigation through the tutorial. Analysing students’ actions, menu selections, reactions and body language on the video recordings were combined with gaze replays, the retrospective comprehension test and the semi- structured, open-ended questionnaire students complete after the eye tracking, to expose usability problems with the tutorial. Improvements to the design could therefore be based on learner interaction and feedback.
Together, these insights were used to refine and adapt the tutorial for further use. During the second cycle, this was used as indications of whether refinement of the tutorial solved or improved usability problems with the tutorial. These insights were once again used to refine and adapt the tutorial for further use.
The findings from the data analysis during the first cycle are reported in Chapter 6 and the findings from the data analysis during the second cycle are reported in Chapter 7. These two chapters also report on how the data analysis was used to adapt and refine the tutorial for the next cycle in the study or for further use.