6 . 4 - 6.1 - < O . 5 . 8 -\ 5 . 5 - 5 . 2 - A
\
c
V
AS\
Figure 5.5 Main effects o f factors on the performance variable (log of time to com plete ‘identify or ‘distinguish’ tasks): Left) Main effects o f the visual operator. Centre) Main effects
o f the position in which tasks were administered. Right) Main effect o f subjects.
Before we analyse this last point in detail, we m ust check how m uch m ore efficient was the use o f the Latin Squares compared to completely random ised and block designs. We first estimate the same models as in Equations 4.4, 4.6 and 4.7 but for y , the perform ance variable, being the time taken by users to complete the ‘com pare’ and ‘associate’ tasks. First, we com pare the residual standard error o f the completely random ised and block designs (0.4890) to that o f the Latin Squares deign (0.1656). Replacing these values in Equation 4.5, we obtain that the Latin Squares represented an im provem ent com pared to the complete random ised design o f 295.3%. Second, we com pare the residual standard error o f the block design w hen the position in which the tasks were administered is used as the block variable (0.5365) to that o f the Latin Squares design (0.1656). Replacing these values in Equation 4.5, we obtain an increase in efficiency o f 323.97%. Third, we com pare the residual standard error o f the block design when the subjects were used as the block variable (0.1552) to that o f the Latin Squares design (0.1656). Replacing these values in Equation 4.5, we find that the Latin Squares was slightly less efficient than this block design in 6%. Still, it was considered that the Latin Squares was an appropriate choice o f design. This highlights the relevance o f taking into account the position in which tasks are administered in these experiments.
Chanter 5 Refining Task Characterization
T ab le 5.14 E stim ates o f effects for ‘com p are’ and ‘a sso c ia te’ tasks
Estimate Standard Error t p-value
Mean 5.93 0.09 63.47 0.00 Visual Operator V (A) 0.33 0.16 2.06 0.05 V(AS) -0.46 0.16 -2.89 0.01 V(C) 0.11 0.16 0.65 0.52 Subject 8(1) 0.49 0.25 1.99 0.06 S(2) 0.44 0.25 1.80 0.09 S(3) -0.38 0.25 -1.54 0.14 S(4) 0.28 0.25 1.17 0.26 S(5) 0.36 0.25 1.44 0.17 S(6) 0.21 0.25 0.84 0.41 SÇO -0.57 0.25 -2.31 0.03 P osition P (l) 0.32 0.16 1.98 0.06 P(2) -0.13 0.16 -0.79 0.44 P(3) -0.08 0.16 -0.50 0.62
In summary, these results are point towards an im portant finding, namely that individual differences have an significant role on experiments where visual spatial operations m ust be accomplished. Similar results have been obtained in the field o f cognitive psychology as is discussed in detail in §5.3. Therefore, we turn now to analyse user strategies as a means to understand some o f these differences.
5.3
U s e r S t r a t e g i e s
In the experim ent discussed in Chapter 4, it was found from interviews and a Likert scale questionnaire that the system evaluated was perceived by participants as being both useful and easy to use for accomplishing visual exploratory tasks. However, Sutcliffe et al. (2000a) found that while visual user interfaces for inform ation searching m ight be usable, they may no t help users achieve im proved perform ance. User perform ance in the second experim ent was analysed using recorded measures o f the time taken by users to com plete each task and the num ber o f successful answers during the two testing sessions. It was found that user perform ance im proved during the second session because o f the previous experience with the software environment. It was also found that, across all users and tasks, 80% o f aU answers were correct, or m ore precisely, within the range o f plausible solutions. 77% o f respondent answers during the first day were correct and 83% during the second, suggesting an im provem ent. These findings are in accordance with those o f Kyllonen et al. (1984a) w ho suggest that subjects w ho have a high degree o f proficiency in a particular dom ain perform best if their strategies and skills are sharpened through practice and feedback. By high proficiency in a domain, we m ean in this case that they were all average to expert users o f GIS and data. In terms o f feedback, users had an introductory or training period before each
('haprer 5 Refining Task Characterization
experim ent started, during which they could ask any questions including those concerning the identification o f ways o f finding particular inform ation. Practice, on the other hand, was gained over the course o f the two experiments
In addition, following observation o f user behaviour and from the interviews carried out during the experim ent reported in Chapter 4, it was hypothesised that participants were employing different strategies in order to reach their answers. A num ber o f studies show that subjects differ in the strategies they use and often shift strategies between and within spatial tests to accom m odate different aspects o f tasks that they m ight find difficult (Kyllonen et al.
1984a; Kyllonen et al 1984; Snow 1978; C ooper 1980). Kyllonen et at. (1984: 1325) cite Snow (1978) w ho points to three possible sources o f individual differences in task perform ance which are loosely referred to as strategy differences: “sequence differences, where subjects differ in the steps that are executed; route differences, where subjects differ in the steps that are included; and sum mation or strategy differences, where the entire processing program differs from subject to subject or differs within a subject for different items o f a task”. These three possible sources o f individual differences were investigated for the third and last set o f experiments.
5.3.1 U s e r B e h a v i o u r
In an attem pt to understand the strategies used by subjects in the context o f geovisualization, respondents’ mental and physical operations were studied (Sutcliffe et al. 2000; 2000a). Users were video taped and the tapes analysed to investigate their behaviour through the operations they perform ed with the software. These operations are defined in Table 5.15. In order to solve a task, users had to perform a sequence or com bination o f low-level actions such as searching and selecting inform ation. W ith the functionality provided by the environm ent, this could be done either using the GIS (map) or any o f the other non-spatial graphical displays. O ff-the-shelf GIS functionality offers selection by attribute and by location, as well as colour coding o f areas for representing attribute values. B oth on the map and on the rest o f the visualizations, users could interactively select objects by enclosing them in a bounding polygon. In addition, data filtering using query devices was available for concentrating on subsets o f data and emptying all views (apart from the map) from unwanted clutter.
Physical Operations (Observed)
1. E xecute a search by attribute Query the database using the search by attribute dialogue w in d ow in A rcG IS
2. E xecute a search by location Send a spatial query to the database using the search by location dialogue w in d ow in A rcG IS
3. Filtering Send an SQ L query to the database using the filtering devices
(Query D evices) in D ecisionSite
4. H ighlight or navigate records Brow se or hover records or spatial features
5. Select records interactively D raw a bounding b o x around records to be selected in any o f the graphic displays
6. Evaluate content Scan a record’s attributes using the D etails-on -D em an d
w in d ow in D ecision S ite or the Inform ation T o o l in ArcG IS
7. Manipulate map Change colour coding or layers displayed
Mental Operations (Observed or 8. D ecid e threshold
verbalised in interview)
D ecid e the criteria or variable values o f relevance for solving a task
9. Select records C h oose subsets o f data to investigate
5.3.1.1
Frequencies
Table 5.16 shows the num ber o f times users perform ed these operations for each task. They were extracted and com puted from the recorded logs and from observing the videos. N ote that the ‘decide threshold’ operation has been com puted by summ ing all activities that entailed identifying a m inim um or maxim um value (or both) when searching inform ation within a subset o f the data. This includes user actions such as perform ing a ‘search by attribute’ in the GIS and operating the query devices for ‘filtering’, which are both equivalent to perform ing structured queries. Similarly, the ‘select records’ operation has been com puted by sum m ing all selection activities such as selecting from any graphical display interactively or ‘selecting by location’ in the GIS.
Table 5.16 suggests that these two sets o f operations, namely ‘decide threshold’ and ‘select records’, account for 77.5% o f all the user actions. 36.4% o f the operations across aU tasks consisted o f interactive searches and selections while 41.8% consisted o f filtering using the query devices. Given that aU participants were GIS users and that they had no prior experience with visualization tools, this latter result is a very high proportion o f the operations which are not perform ed using the GIS. In addition, note that searches by attribute values in the GIS and filtering using the query devices are equivalent operations, namely sending a structured query to the database. However, users m uch preferred using the
Refining Task Characterization
query devices. Participants expressed the view that the query devices allowed them to assess the sensitivity o f the outcom e to the value thresholds im posed by the structured query. In addition, users considered the devices supported their thought process as their effect on the data explored could be seen in near real time. It is through immediacy o f the visual feedback generated that the query devices allow users to explore and assess their ideas as they have them. This is therefore an indication o f the ease with which data could be explored using visualizations other than the map display, such as the PCP, scatterplot or query devices. It also points to the usefulness and benefit o f com bining spatial data displays with flexible, interactive and rapid data query, as well as with aspatial data views that represent different relations present in the data. O ther studies have also reported the benefit o f these features for inform ation search and exploration (Sutcliffe et al. 2000; Ahlberg and Shneiderman 1994; Andrienko et a l 2002).
Table 5.16 Frequency of user operations from computer logs and videos
IN