Research design
3.4 Research methods
This chapter reports a prototype evaluation in which five academics and twenty six students’ use of MLCAT and podcasts respectively were tracked for approximately eight weeks. The evaluation acted as a dual-purpose research vehicle to satisfy two high-level objectives:
1. To evaluate the MLCAT prototype presented in Chapter four.
2. To build on the “pilot” findings reported in chapter four by investigating podcasting over an eight week period.
In terms of both objectives, the study makes important progress over previous work. Firstly, published literature (see chapter three) indicates that there exists challenges such as limited guidance on the methods used to evaluate podcasting and the sequencing of these methods. The next two sub-sections discuss each objective in turn.
5.1.1 Objective 1: MLCAT Prototype Evaluation
Robson (2001) defines evaluation as an attempt to assess the worth of an innovation or intervention. The evaluation of interactive designs is an essential component of HCI research, since an interactive artifact, however innovative, does not in itself constitute substantial contribution to HCI knowledge without some assessment of its worth (Dix et al., 1997; Carroll, 2000). However, much of the body of podcasting design-based research is limited in this regard.
In fact, chapter three highlights the challenges of evaluating podcasting designs. Traxler and Kukulska-Hulme (2005) reveal that the last decade has seen a dramatic increase in the number and the variety of pilots and trials involving m-learning. This increase is reflected in, and supported by, submissions to mobile learning conference series. The growing pedagogic and technological sophistication of mobile learning pilots and trials is evident, although sustained deployment will depend on the quality of analysis and evaluations (Traxler and Kukulska-Hulme, 2005).
106
Evaluation and analysis are essential to the sustainability of podcasting because firstly, they inform the outside world about the effectiveness of pilots and trials with respect to objectives set out initially. Secondly, evaluations provide funders with insights on the cost-effectiveness and utility of pilot projects. In other instances, evaluation determines whether pilots can be extended into longer term Institution-wide projects thus creating sustainability (Traxler and Kukulska-Hulme, 2005).
Therefore, a starting aim was to avoid such limitations by evaluating the design presented in chapter four using a triangulation of methods. MLCAT is a podcasting application which allows a user to author and distribute lectures for download and sharing via Bluetooth. A PAR incremental design approach was used to enable undemanding incorporation of core podcasting processes into a prototype. Chapter four concluded with an initial assessment of the workability of MLCAT based on results from usage by faculty and students at TSiBA – a private HEI in Cape Town, South Africa. Based on the positive feedback from the three academics and twenty one students (i.e. study D), it was decided that the design was feasible, albeit with some minor improvements in functionality i.e. re-designing the client system; automating the distribution of recordings and re-designing the Web-based SnapAndGrab system (see chapter four, sub-section 4.5.3). In effect, no new designs were evident at this point thus validating the original MLCAT design. Consequently, further evaluation was pursued after these alterations had been implemented. The deployment and evaluation had two sub-objectives:
1. To facilitate the formative redesign of MLCAT.
2. To explore appropriate methods for designing and evaluating podcasting tools.
The first sub-objective was to assess the usability of MLCAT and identify ways for its improvement. The specific areas investigated included the following:
Usefulness: do users value the ability to author podcasts in an automated way?
Learnability: how easy is it for users to understand or learn and use MLCAT?
107
How do users respond to MLCAT? – The exploratory study in chapter four identified a wide range of user profiles. A key interest was to examine how users react to MLCAT (e.g. what they choose to record, how students appropriate recordings, etc.).
Secondly, as well as assessing the MLCAT design specifically, it was also hoped that the evaluation would allow the derivation of general design recommendations for podcasting systems. Finally, it was envisaged that lessons learned during the study would provide insights regarding appropriate methods for the evaluation of podcasting tools.
5.1.2 Objective 2: Empirical MLCAT Evaluation Study
As well as evaluating MLCAT, the study also offered the opportunity for further empirical investigation of podcasting. In the context of the thesis, the MLCAT field study provided an opportunity to build on the “snapshot” exploratory studies reported in chapter four. It was envisaged that collecting evaluation data would provide insight into the following issues:
How is podcast content/media used by students? What are lecturers’ and students’
perceptions of podcasts/podcasting?
During “pilot” studies, participants provide subjective reports of how they perform such sporadic tasks. It was hoped that the field trial would enable the collection of more objective data on these aspects of podcasting.
5.1.3 Contributions
The contribution of this chapter towards the thesis is two-fold, based on the dual-purpose nature of the study:
–
Firstly, the chapter offers results from the formative evaluation of MLCAT. In addition, the chapter provides empirical ground-work for deriving general guidelines for the design and evaluation of podcasting tools. These are discussed in chapter six, along with methodological recommendations for designing and evaluating podcasting tools based on the experience gained in evaluating MLCAT.Secondly, the chapter offers insights into the nature and use of podcasting.
108
Section 5.2 describes the study timelines and methods; section 5.3 presents findings while section 5.4 presents a discussion.