Technical Setup

For both conditions used in this study, students used PaSAT client software running on Mio Pocket PC PDAs with built-in GPS for location tracking. As described in Chapter 4, the client software on the PDA connected to a server application running on a laptop via a wireless network connection. The PaSAT server laptop was connected to a dedicated wireless router to provide wireless coverage in the learning space. The connection was via Web Services and hence stateless Ð if the connection is dropped temporarily there is no immediate impact on learner activity (see Chapter 4). The system also used a thin-client design whereby all session data is stored on the

server and not on the PDA Ð if the PDA needed to be restarted no data is lost for that session.

5.2.6.1 Outdoor Condition

For the outdoor condition, the PaSAT server was deployed on a laptop connected to a wireless access point, with power provided to both from inside the school building via a 50m extension cable. Following initial trials that had indicated problems with extending wireless coverage to the area required for the learning activity, the access point was fitted with an additional antenna to boost its range.

5.2.6.2 Indoor Condition

Students used the same PaSAT software running on PDAs connected to the server via a wireless network, but they used the PDAs indoors and indicated their location by clicking on the screen and not by moving around. The same server-client setup was used, with the laptop server located indoors in the same room as the students.

5.2.7 Evaluation

We used a number of different methods to observe, explore and explain the activity of the learners during Study 1.

5.2.7.1 Video recording and direct observation

In line with studies of similar PDA-based learning activities (for example, Facer et al., 2004; Squire and Klopfer, 2007), we used an observational approach and then reviewed video data to look for evidence that related to the research issues outlined in Section 3 above. In particular, in line with Squire & Klopfer (2007), we focused our attention on unexpected factors.

The observation notes and video footage were reviewed for episodes related to the issues being investigated. Any other significant episodes that were not related to the

research questions were also flagged for further analysis, in line with the critical incident technique (see Chapter 4).

5.2.7.1.1 Outdoor condition

The studentsÕ activities and behaviour during the session were analysed using video recordings made during the session and the direct observations made by the observers. Students each had a PDA to use during the task, but were asked to work in pairs and to stay as close as possible to one another. An observer with a video camera followed each pair and observed and recorded their activity.

Observers were also requested to flag any notable critical incidents by moving their hand in front of the camera, and to report any significant events after the session.

Unfortunately two of the five cameras malfunctioned (one hardware failure, one battery failure) during the session, so only three tapes were available for analysis.

5.2.7.1.2 Indoor condition

For the indoor condition, students worked in a classroom, each using their own PDA. For the purposes of video recording, they were grouped into three groups (two groups of three and one group of four students). The analysis was based on reviewing the footage from these three tapes (only three cameras were available for this condition following the camera malfunctions in the outdoor condition).

5.2.7.1.3 Critical Incident Technique as used for this study

To identify specific aspects of the outdoor location-based activity that led to either breakdowns or breakthroughs in learning (Sharples, 1993), we employed a modified version of the critical incident technique (Flanagan, 1954). The critical incident technique (CIT), and its applicability to this work, is described in detail in Chapter 4.

We modified the technique to fit with the limited time we had to work with the students. Instead of reviewing each critical incident with the participants, we used the

video footage to perform an in-depth analysis of each critical incident. After a critical incident had been identified, it was reviewed on the footage, taking note of the context, causes of the incident and any impact it had on the task activity.

This process was started before the focus group with the students took place, so that we were able to structure the questions in the focus group to probe specific incidents. This particular modification of CIT has been employed before in exploratory studies of learning technology, for example Anastopoulou et al. (2008).

5.2.7.2 Pre- and post-task quizzes

Students were asked to complete pre- and post-task quizzes (see Appendix C) to assess their recall of the content encountered during the task. The quiz comprised a series of questions relating to flooding and flood defences, with an open answer format. An open answer format was necessary because of the limited amount of content presented during the task: it would have been impossible to produce pre- and post-task quizzes that used different questions.

The questions were devised to test studentsÕ knowledge of types of flood defences, both before and after completing the learning activity. Students were asked to provide examples of types of flood defences along with advantages and disadvantages for each. This mapped on to the content provided during the activity.

The aim of the quizzes was to provide an indication of whether there were any directly observable differences between the studentsÕ learning in the outdoor condition compared to the indoor version.

5.2.7.3 Post-task map drawing and annotation

Students were each asked to draw on a map of the area with the locations of content hotspots and with notes describing what each hotspot related to. The students were

also asked to describe (in note form) any memorable incidents from the task, for example Òwe made a joke here about what the tree looked likeÓ.

The purpose of these maps was twofold. Firstly, we wanted a quantitative measure of how well the students recalled the layout of the map and the features on it, compared between the two conditions. Secondly, we used the maps as a way of determining whether the students had any underlying conceptual misunderstandings arising from the task. By asking the students to make notes about what they had found, we gained some insight into their understanding of the task over and above the pre- and post-task quizzes. The intention was also to determine what aspects of the task were memorable for the students, and to provide them with a way of feeding back narrative descriptions of what they did.

5.2.7.4 Changes to chosen evaluation methods following field trials

After conducting the indoor learning activity and reviewing the footage, it became clear that it was difficult to identify critical incidents for the indoor version, as had been possible for the outdoor condition.

This was due to a number of factors:

¥ Low visibility of student activity: students were seated at a table each using a PDA. Video footage was from the front of each student, which meant that their actual activity was not as visible as was the case outdoors.

¥ The task was not as engaging, leading to fewer observable events in general.

¥ The activity was conducted within a single room.

¥ Low levels of activity compared to the outdoor version.

Following this observation, we decided instead to focus primarily on the critical incidents identified for the outdoor condition, and for each of these to then review the

footage from the indoor version to see if we could compare directly between the conditions. Where direct comparisons were possible, this is shown in the analysis of the critical incidents shown in 5.3.2. Where this was not possible, we instead reflected in general terms on the nature of the indoor condition compared to the outdoor one.

5.2.7.5 Post-task interviews

As well as completing the post-task quiz materials described above, all of the students who took part in the outdoor activity were interviewed in a group to gather their opinions on the activity. They were encouraged to express both positive and negative opinions, and it was emphasised that their input would contribute to improving the system for subsequent use. The interviews used open-ended, semi-structured questions to identify key issues related to the task and explore them with the participants. In some cases critical incidents identified from the task activity were related to the participants to prompt discussion. The researcher took notes during this session.