Summary of end user evaluation results

The end user evaluation was undertaken in two settings, with ten participating in a facilitated group in an academic setting and twenty-five participating online non- facilitated survey. Observations and feedback from the group session revealed that that clear instructions were essential to avoid confusion in the set up and operation of the virtual globe tour.

Scale questions in the end user survey were rated 1 to 6 and designed to obtain feedback on the usability of the tour, the effectiveness of specific elements, and the helpfulness for communicating both the features and issues of the catchment and the potential for NFM in the catchment. The survey also asked about viewers perceptions of the impact of NFM on the environment.

Usability ratings in both settings were high for ease of use, visual clarity and level of trust, although they were slightly lower in the online survey. The mean ratings for these metrics were above 5 for the facilitated group. The mean value for level of trust was also above 5 in the online survey, whereas the ease of use and visual clarity were above 4. The ratings for presentation speed were lower, (mean values of 3.4 in the group and 3.2 in the online survey). This may indicate the participants found the tour slightly slow or reflect that the rating scale had no neutral rating.

The mean ratings for effectiveness of the tour elements were all high, above 4 in both the group and online surveys. The lowest ratings were for the close

up/perspective views (mean = 4.9 group/5.0 online) and the visual appearance of surface features (mean = 4.6 group/4.3 online). The highest ratings were the pop- up information balloons (mean = 5.5 group/5.1 online). The mean ratings for effectiveness were consistently higher (and the variance lower) in the facilitated group than the online group for all the elements.

Comments made in free text questions supported the usability metrics, indicating the tour was easy to use - “clear and easy way to present a lot of information”, novel - a “fun way of learning”, with clear instructions, although several referred to issues with visual appearance of flashy lines and images during movement (n=6) and the slope cross sections (n=3). Participants commented that they liked the pop-up balloons (n=2), the interactivity with links to further information (n=4) and the ability to pause and rewind (n=4) in addition to the fly through navigation.

There was some disagreement on the level of interactivity with one noting there was not enough and another that many of the links were not relevant to them. Only one participant referred to the Google Earth imagery noting that it did not give an adequate view of the topography.

Among the online participants there was no notable relationship between age, gender, occupation or experience levels for Google Earth or computers with the rating of ease of use or visual clarity. Those with higher Google Earth experience or interest in GIS did rate the speed of presentation lower and those with GIS interest also rated the level of trust in information lower.

Mean ratings for helpfulness were all above 5 in the group setting survey and above 4 in the online group. The highest ratings were for demonstrating the catchment features (mean = 5.7 group/5.2 online). The lowest ratings were for the understanding of NFM techniques (mean = 5.2 group/4.2 online) although the ratings for demonstrating the potential for NFM were higher (mean = 5.4 group/ 4.9 online). Ratings were also high for identifying sources of information (mean = 5.4 group/5.0 online).

Free text comments supported the survey ratings that the tour was helpful for communicating the catchment features including the topography (n=6), the

ecology (n=4) and historical environment (n=2), with several referring to being able to see the flood area (n=8). They also commented on the issues of land

management change (n=6), the complexity of multiple agencies (n=3) and the potential for using NFM (n=2). Others did however note that they would like more photographs to improve their understanding of the area (n=5) and had expected more specific information on NFM techniques and where they could be used in the catchment (n=2).

In the online survey there was a noted difference in the rating of helpfulness for demonstrating NFM techniques according to the participants levels of prior knowledge. None of those stating they know a lot about NFM rated it above 5 whereas 71% of those knowing nothing, and 45% of those knowing a little, rated it above 5.

The sample size was not large enough for detailed analysis and not all categories were well represented (such as those aged under twenty, resident in the area or personally affected by flooding). However, there was an indication that the rating

of helpfulness for demonstrating the features and issues of the catchment was higher among those with some familiarity of the area; a higher proportion of participants who visit occasionally rated it highly (5 or 6) for demonstrating the features (100%) than those who didn’t know the area (64%).

The results showed that after viewing the tour participants perceived that NFM would be beneficial to flooding (100% in the group and 96% of the online participants) and to water quality (88% in the group and 90% of the online

participants). There was more variance in the perceived impact on landscape, soil quality, farming and wildlife. 21% of online participants selected ‘Don’t know’ for the impact on farming and 16.5% believed there would be a detrimental impact. As these questions were only asked after viewing the tour it was not clear how much the tour had influenced these perceptions although the ratings for

helpfulness had indicated that it was useful for demonstrating NFM for those with little or no previous knowledge.

Only one participant had personally experience flooding to their property and the results showed no obvious difference in the rating of the helpfulness of the tour for communicating features or issues for the catchment or the perceived impact of NFM according to personal experience of flooding.

38% of non-facilitated online participants indicated that they were ‘very’ interested in finding out more about NFM and 58% ‘somewhat’ interested. Of those with some level of familiarity with the catchment area 50% were ‘very’ interested in finding out more compared to just 18% of those who had not heard of the area.