Procedure for collecting data

Group A participants were pre-tested, then exposed to a visual demonstration intended to show them how to use the visualisation tool, which is customised software. The demonstration involved participants ‘seeing and interacting’ with the visualisation but not exploring all possible scenarios. Then they were given guided-exploration tasks using the interactive visualisation tool, ElleVis. Finally, the participants were post-tested.

Group B participants were pre-tested then shown how to locate and interpret the different features of ElleVis using the paper-based visualisation. They were then given the same guided-exploration tasks as Group A, however this group completed them using the non-interactive visualisation rather than the interactive visualisation. (see Appendix F.3). A post-test then took place.

After the post-test, the research design for both groups moved to a qualitative phase – a post-study guided discussion. The design was intended to give participants time and opportunity to think about the forms of information provided to them and to record their own impressions. A series of questions was developed to gather feedback on the issues participants encountered using the visualisations. The focus of the questions included: whether the forms of intervention did or did not assist them in their ability to answer questions in the post-test; the perceived benefits and drawbacks with the form in which information was presented. Each interview was digitally recorded with the permission of participants, and written notes were also made. The recording was subsequently transcribed, allowing for thematic analysis.

The researcher worked with each participant individually at a time and place most convenient to each participant.

9.2.3

Data analysis

Data collected in the pre-test, post-test and post-study guided discussions allowed several types of analyses to be undertaken. These were necessary in order to triangulate responses from different ways of asking how forms of information influence learning and knowledge acquisition.

Participants’ responses in the pre-test and post-test were interpreted against the grading schedule (see Table 6.7) using the mean and standard deviation to determine if there were any changes in their levels of personal understanding. To determine whether differences in scores between the pre-test

independent t-test was performed on the comparative changes in understanding that occurred in both groups to determine whether there are significant differences in knowledge outcomes depending on the form in which information was presented. (See Chapter 10, Section 10.3, for the results of this analysis.)

9.2.3.1 Analysis of qualitative data

Participants were guided in the post-study interview by a series of prompts. The following prompts embedded in questions were used to guide the discussion: usefulness of the visualisation (non- interactive or interactive); most effective component of the visualisation; alternate forms of information provision to assist in understanding of lake issues; prior knowledge of the lake; usability of the visualisation tool (non-interactive or interactive); accuracy of the visualisation in depicting the lake behaviour; increase in knowledge about the lake after using the visualisation; benefits of the visualisation (non-interactive or interactive); other general comments about the way information was presented. (See Figure 9.3 for an illustration). The interviews were transcribed verbatim and the data organised into groups which relate to areas of interest as generated by the ‘prompts’ above.

Data was arranged using the NVivo software based on ‘self-emerging’ themes from the transcripts (Halcomb & Davidson, 2006). These themes were examined in relation to the key research question of the follow-up study. Themes explored include: discussions highlighting on interactivity of the visualisations; effectiveness of the visualisation format; usefulness of the visualisation; ease of use of the visualisation; the perceived benefits and drawbacks associated with the form or forms of information presented (refer to Table 9.1 for an example of the coding system used for the study). The ‘codes’ transfer across interviews and thus allow comparison.

A content or thematic analysis approach was used to analyse the interviews. This approach systematically analyses texts and requires the researcher to generate codes (for example, themes) directly from the data, and continuously analyse these codes whenever new data are interrogated/inspected (Creswell, 1998; Hesse-Biber et al., 2006; Langan-Fox et al., 2004; Patton, 2005). Figure 9.3 gives an example of a portion of a coded interview where the codes identified in column 1 of Table 9.1 are applied directly to segments of the text in situ. (Chapter 10, Section 10.4.1, provides the results of this analysis.)

Table 9.1 Coding system

Code Theme Code Theme

1. The accuracy of visualisations in showing lake

behaviour 5. Other forms of information provision that would also increase understanding 1.1 Mixed responses from participant regarding

perception of accuracy of ElleVis 5.1 Significant ‘other’ forms of information provision 1.2 Negative responses from participant regarding

perception of accuracy of ElleVis 5.2 Satisfied with current provision of ElleVis 1.3 Positive responses from participant regarding

perception of accuracy of ElleVis 5.3 No idea/don’t know

2. Most effective component of the visualisation 6. Increase in understanding of lake management issues after exposure to visualisation

2.1 Summary table 6.1 Positive response 2.2 Line graph 6.2 Negative response 2.3 Spatial map 6.3 Mixed response 2.4 All of them 6.4 No idea/don’t know 2.5 None of them

3. Usefulness of the visualisation 7. Overall general comments regarding visualisations

3.1 positive response suggesting useful nature of

tool 7.1 Positive response 3.2 negative response suggesting tool needs

improvement/lacks clarity/needs amendment 7.2 Mixed response 3.3 Mixed response regarding usefulness of tool 7.3 Negative response 4. Usability of the visualisation tool 8. Interactivity

4.1 User friendly/easy to use 8.1 Describing interactivity 4.2 Rigid 8.2 Commenting on interactivity 4.3 Suggesting improvements to make tool user

friendly

4.4 Don’t know/no idea 9. Prior knowledge of lake 9.1 Work/stay around the lake 9.2 Affected by the lake in many ways 9.3 Other interests in the lake

Interviewer (I): Do you think this computer programme has enhanced your understanding of some of the lake management issues?

Participant (P): Yes definitely 6.1. I think it does 6.1. It has a nice visual input 3.1, and the interactivity 8.2 for me is well collaborated 4.1. Because for me I can look at the lake level and look down at the colour coded information which is really good 3.1, 8.1.

I: Do you think there is any other way information about the lake could have been presented to you that would increase your understanding

P: I will find it very hard to improve upon what you have presented 5.1. I think yours is a very good presentation 3.1 and as I have said previously probably it improved my understanding 6.1. When it came to the test questions again, as I was able to match 8.1 the graph level with the months of the year so I can be looking at the optimum there and when the lake was opened and when it was closed 4.1 and I had a better way of relating 8.1 that to the issues I was comparing

4.1.

I: Oh ok. Do you think this computer program provides an accurate picture of how the lake behaves?

I: So do you think that this computer programme that is provided to you increases your understanding about your knowledge about the lake?

P: Yeah it does 6.1 because I have lived here for a long time 9.1 and did not understand about the opening and closing and this helps me to understand 6.1, 3.1 that the opening and closing is dictated to by specific levels and did not realise that ECAN controlled it closely by actual measurable levels. I thought it is more of a rattle reading that if it is pretty high, let’s open it up. So it has contributed to my understanding 6.1.

I: Which component of the visualisation did you find most effective?

P: The summary table 2.1. The colour coding is well structured and easy to interpret 4.1.

I: So do you have any additional comments you will like to add. I mean any general comments about how information was presented to you today?

P: I found that when information was presented to me in the beginning I was not good at looking at numbers and understanding what the process is but having the information laid out actually 3.1 and how they interact 8.2 with one another 4.1 and the change this scenario and you could look at graph of water level and compare down to how it impacted 8.1 on the population is pretty good and pretty useful 7.1

Figure 9.3 Interview segment with appropriate coding inserts: An illustration

9.3

Summary

This chapter has described the research design conducted to determine whether there are significant differences in knowledge acquisition depending on the form in which visualisation of environmental changes is presented. It was anticipated that the results would provide greater certainty that the changes in knowledge demonstrated in the initial study were indeed the result of using computer- aided interactive visualisation.

All participants were members of the public who have a general interest in the management of Te Waihora/Lake Ellesmere.

A split-sample test was used to communicate the same information about the lake but in two different forms. One group – Group A – experienced the interactive visualisation while the other – Group B – experienced screen shots taken from ElleVis as a non-interactive/paper-based visualisation.

A variety of data were collected in the pre-test, post-test and the post-study discussion. These data were collected to determine whether there were any changes in the level of personal understanding of participants regarding the impact of opening regimes on lake values at Te Waihora/Lake Ellesmere and whether changes in understanding that occurred reflected the forms in which information was presented. The following chapter, Chapter 10, presents the quantitative and qualitative results of this follow-up study.