Purpose of Workshop 88 

In fulfillment of two of the research questions introduced in Chapter One, the purpose of this workshop was twofold. The first objective was to test SunSpot’s effectiveness in communicating local level climate change issues and assisting different user groups in associated decision making processes. The second objective was centred on identifying recommendations for future improvement of SunSpot and similar Web-SDSS applications. As an added result of this exercise, the application was also tested for technical issues that may have been overlooked during internal testing. These objectives were met through a series of in-person workshops that guided participants through the use of SunSpot in order to observe activity as well as solicit written feedback. Participant feedback was shaped through

participants’ perspectives and to provide a medium for communicating identified problems and needed features that might be satisfied in later development.

4.1.1 Usability Engineering

Usability is the measure of how well an application fits the needs of its intended audience and how well it enables them to complete desired tasks (Landauer, 1995). Usability engineering (UE) investigates the development of common practices that optimise an application’s usability. These practices involve recurring themes and design principles that have been shown in previous work to be effective in addressing the usability of an application (Butler, 1996). Haklay and Zafiri (2008) reference one list of concepts using the ‘five E’s’ of UE: efficient, engaging, effective, error tolerant, and easy to learn. The authors believe that UE for GIS is becoming increasingly important, where elements such as these require careful consideration when designing applications to meet user needs.

Research has been conducted on how UE applies to Web-GIS development with many authors discussing how compatible traditional methods are in this context. UE is considered to be of great importance due to the interactive nature of Web-GIS and other geovisualisation applications (Tobon, 2002; Schmiguel et al., 2004; Haklay and Zafiri, 2008). In addition to efficiency and engaging (described as ‘interactivity’), Sidlar and Rinner (2006) identified additional issues to be evaluated that are unique to web-based applications, including cost of entry and Internet connectivity. Cost of entry is with regard to potential barriers such as software requirements on both the server and client side that may impede users to be able to access the application, or authors from keeping it available. Connectivity, such as bandwidth, latency and persistence of connection between the client and the server can impact accessibility of some Web-GIS applications, such as those designed for consistent bi- directional data flow or to share large volumes of data.

In addition to testing the technical components that relate to usability, it is also important to investigate how well users understand and value the data and results obtained from using the

application. This is either considered as a by-product of UE testing and therefore investigated as a component of user satisfaction and memorability, or investigated separately by explicitly testing for knowledge and understanding developed by the user (Koua et al., 2006; Sidlar and Rinner, 2006). These authors used numerical scales to poll participants on their satisfaction and understanding of results. In cases where the results are more complex, further investigation including in-person observations, discussion of the results, and written discussion on user perceptions is important.

In measuring the success of UE, a number of authors consider many traditional methodologies to be ill-suited to Web-GIS and geovisualisation. The nature of GIS and, in particular, SDSS involves the exploration of problems that are often ill-defined in scope and methodology (Keim, 2001). The application’s function is often directed at exploratory tasks or the investigation of dynamic issues where the user acts as a supplier of data or expertise (Andrienko and Andrienko, 2006; Tobon, 2005). Many UE methods test the ‘five E’s’ in a manner that does not assume an open-ended use. For example, performance testing observes how long users take to complete a specified task, providing an objective measure of efficiency (Dumas and Redish, 1999). Geovisualisation applications often offer numerous possible workflows for users to analyse and investigate intermediate results, make adjustments, and test different inputs or assumptions. Attempting to gauge usability of these tasks by measuring the time it takes to complete them would therefore be meaningless.

There are a number of ways to test geospatial applications that authors have shown to be effective. Usability testing is one of the most common methods where participants interact with the application while actions and observations are recorded (Dix et al., 2004). Recording can be done in a number of ways including in-person observation by facilitators, software data capture of interactions, or video recording of the computer screen and, in some cases the participant (Lin et al., 1997). In-person observations provide a holistic sample of HCI as the facilitator can observe both the screen and user interactions in real-time, including observing

sessions provides similar capabilities, but is lossless compared to human memory and permits many more participants to operate simultaneously. However, this requires more planning, additional costs for technology, and more time-consuming data analysis. The use of eye tracking in UE has been relatively new in the past few decades, becoming more popular as technology capabilities and costs improve. Coltekin et al. (2009) used an eye tracking apparatus to measure the visual movement and focus of participants as they conducted a number of simple tasks on two online map interfaces. Their results found that in combination with other data gathering methods, eye tracking enhanced the usability studies. Software recording of interactions can help reproduce part or all of each user session, allowing facilitators to build reports on how specific UI elements are interacted with, processing times for tasks, and easily identify and reproduce errors (Lin et al., 1997).

The use of questionnaires, often as a supplementary component to usability testing, is also a common method for obtaining feedback. Questions can be formulated to investigate any objective or subjective issues that the user could respond to at any point in the application’s development process (Babbie, 2006). While some questionnaires may be included in, or follow usability testing, others can be used to reach a much broader audience in an affordable manner, especially if the application is Web-based (Haklay and Zafiri, 2008). Questionnaires are also an ideal way to quantify user satisfaction with various components of an application, which is an important component of measuring usability (Chien et al. 1988).

4.1.2 Testing SunSpot’s Usability

An in-person workshop was selected as the optimal format for initial tests of SunSpot. While SunSpot is a Web-based tool and could immediately be accessed by participants remotely, an in-person workshop helps ensure that the testing methodology is effective before exposing the workshop to a wider audience. This provided the greatest flexibility of interaction with the participants as any potential roadblocks in testing would be quickly identified and alleviated. In this sense, an in-person workshop also helped identify how effective the workshop is in exposing participants to all components of the application and soliciting

valuable feedback. This lays the groundwork for possible future testing that may take advantage of a wider audience. A substantially larger test group from online testing may provide little useful feedback if the workshop design or application itself has critical errors that inhibit use.

The in-person workshop provided an opportunity for personal interaction, which allowed additional feedback to be gained through informal discussions as well as brief ‘over-the- shoulder’ observations. Discussing different components of the application with users allowed valuable data to be collected that may not easily be expressed through other forms of response, such as the strength of opinions regarding certain components. It also provided an opportunity for open-ended discussions about parts of, or the whole application that may not be possible through written responses. Observing how participants performed tasks in real- time helped identify issues that may not have been considered when designing the written survey. Not only did this shed light on further issues to consider, but it helped shape the informal discussions that took place during and after the workshop. By combining different methods of data gathering, confidence in the results could be improved. Known as triangulation or cross-examination, this method increased the data points gathered while mitigating potential bias introduced from using a single technique (Creswell, 2008).

During the design of SunSpot, a module was developed that would record all user interactions with the application. This module allowed each mouse click along with screen coordinates and time to be recorded. In addition, the manipulation of all tools including changing the map view, toggling visual layers, opening and closing widgets as well as executing widget workflows would be recorded. As SunSpot has no stochastic components, these data would be sufficient to allow sequential playback of a user’s session and the ability to ‘step-through’ each action at any pace. While part of this tool was designed and programmed, completion of it was abandoned due to increasing complexity and time constraints. Such a tool, while providing highly detailed observational data, would require