Piloting

Piloting for any research project is extremely important and enables the researcher benchmark the chosen methods against the studies aim. In this case establishing

appropriateness, quality and accuracy of the procedures adopted to support the research questions (Kumar, 2011). The pilot study ran for twelve weeks starting 01/10/12 this gave the researcher time to reflect on methods and data collection in preparation for the ethnographic fieldwork which commenced September 2013.

The Pilot aimed to:

 check that the methods were suitable and not intrusive.

 check that the exercises meet the students experience and module learning outcomes.

 make sure trust would be built up between the researcher and the participants.  see if the barrier between lecturer and student was there when an observation took

place.

In September 2012 mobile test frameworks and core aspects of the systematic review (i.e., HIB and Mobile Informatics) were introduced to inform learning and teaching practices. At this point there were some key themes and features emerging from the grounded work around user context, information needs, environments and modality. These themes were firmly set within the researcher’s vision and a core contributor to the aim of study and the outcomes of the module. With this in mind the researcher ran formative sessions over a three-week period where different aspects were used and applied. The researcher simulated participant observations so feedback could be gained on the students’ ability to use these practices to support test design.

71

The feedback was interesting, an important point to note from the evaluation of the ethnography had synergies to past research practice, Cooper et al. (2004), for example, state that the ethnographic texts suggest that judgement is not bound by formal scientific rules, but rather that it is more experience and intuition (p. 5). As an academic with lesson plans and learning outcomes the intuition and experience helped to support observation practice within the ethnographic context. Piloting supported and reaffirmed the modules learning outcomes and based upon this experience and using the researcher’s professional judgement a new supporting point was included in the assessment “Research and develop experiments based on Information Model provided”. This came out of the pilot

observations which makes it clear to the students that test cases (or scenarios) are informed by a mobile test model or framework, which improved support and contextualised

assessments based upon the modules learning outcomes.

Another noteworthy point from the piloting exercise related to the research suitability and linked to the students’ ability to critically think and develop mobile test based upon these models and frameworks - a core learning outcome. This point also relates to context and modality. The pilot highlighted a need for guidance on definitions of contexts which seemed too abstract and the researcher initially thought this could impact their mobile tests. The researcher was initially worried that the disparate nature of their context choices from the pilot study could have an impact on future summative work and research data, for example, “Could the varied and wide ranging context and modalities make it difficult to identify themes and codes for the data analysis?” (student quote, 2012)

There was a worry that there are too many context options and micro-environments that impact on the mobile tests. One solution would be to narrow or prioritise options, which would help or maybe force students to develop tests based around one contextual factor and environment (i.e. wandering down Northumberland street before heading to University). These concerns can be traced back to earlier HCI studies, Beck et al. (2004) had a predefined route with potential obstacles which limited the environmental contexts. This is something the researcher did consider but felt was restrictive and limited the “contextual” nature of this research where physical environments play a big part in contextualising mobile tests. As the research evolved through the systematic review the use of information models in this case Wilson’s model provided a framework of practice to support students’ reducing confusion making the test process adaptive and structured which supported the test need and setting. Based upon this finding the workshop “Using a module to improve testing agility” emerged which helped to shape the students understanding and supported the researchers formative observations sessions.

72

Trust factors are fundamental to any ethnographic research (Cooper et al, 2004; Mcneil and Chapman, 2005), trust is seen as a potential barrier between the students and the

researcher. Students were initially wary about, as they put it, the lecturer “snooping over my work” but once the process was explained and the researcher got to the bottom of the main issue, “is this going to affect my mark?” they did not mind. This is a point that is taken very seriously and the assessment guidance included clearly “Please note. Matt will be out observing students between week 7-10 as part of the fieldwork for his

Professional Doctorate. The class will be on but he might not be there in attendance each week. It is also worth noting that Matt will be observing you as a group between weeks 4 and 6, he will also observe you individually conducting your tests – this is NOT part of the assessment and Matt observing you has NO bearing on your assignment and mark!”

The pilot also explored student data collection which was initially a diary which was going to be used from two perspectives, the students’ and researcher’s perspective. The student diary aimed to follow a systematic approach throughout the module (from weeks 3-12) each week students are encouraged to fill out a diary about their experiences and this would help to support their justifications within the final report. Students were encouraged to gather data reflecting on their practice and how the model supported their tests. This seemed like a good data collection mechanism but based upon the pilot students did not engage in capturing data (even though it was part of the assessment) and it was removed as a final data collection method. The researcher log/diary however did work and will support the

researcher’s attempts to capture attitudes and feelings toward the process. The diary helped to capture points about the model and how it is going to be applied.