Thesis structure

To convey the structure of this thesis, each chapter is described very briefly at first and

summarised by the thesis map in Figure 1, the contents of each chapter are then

described in more detail. Chapter 2 reviews the literature in the fields related to this

work. It describes relevant background material and lays out the evidence to warrant the

choice of techniques used later in the thesis for design and evaluation work. Chapter 3

reports two exploratory studies of document use in a corporate environment which

sought to assess the feasibility of using paragraph level metadata. Chapter 4 describes the design of GridVis, an interactive visualisation of paragraph-level metadata to

support goal-directed search. Chapter 5 reports user trials, task analysis, and

experimental evaluation that were used to assess and further refine the design of

GridVis. Chapter 6 describes the best-case analysis used to ascertain the potential performance of the interactive visualisations of paragraph-level metadata as embodied

by GridVis. Chapter 7 reports the results of this evaluation, and Chapter 8 ties together the conclusions and contributions of this research and suggests avenues for further

work. Figure 1 illustrates the way each of these chapters fit into the thesis as a whole, and locations of the various contributions enumerated in section 1. The content of each

Chapter 1 section 2 33

G

Background investigation Evaluation

Ch 2 - B a ckground Literature

M eta data literature Inform ation seeking Inform ation visualisation Inform ation foraging Evaluation m ethods Related w ork

Ch 3 - C o rp orate d oc um ents and how m etadata could fit in

W riter study

interview s 7 p a rticip a n ts

R eader study

inte rview s 4 pa rticip a n ts C h a ra c te risa tio n of d o c u m e n ts u s e d C h a ra c te risa tio n of d o c u m e n ts su itab le for e n ric h m e n t with p a ra g ra p h -le v e l m e ta d a ta Investigation Into n u m b e r th e g o a ls th a t a re m e ta d a ta e x p re ssib le Design a nd prtxluction

Ch 4 - G ridV is C o nstru ction

Selecting the docu m en ts M eta data Design and Production M eta data Taxo nom y D esign and P roduction

Interview with 8 p a rticip a n ts, k P ro v id e d inform ation o n inform ation

n e e d s a n d w hat is n e e d e d to d isc n m in a te rele v an t m a teria l

Selecting the docu m en ts V isualization Design

Ch 8 - C o n clu sio n s and further w ork

C o nclusions and contributio ns Further w ork_____________________

Ch 5 - Initial e v a lu atio n

U ser trial

4 p a rtic ip a n ts • 3 re a d in g ta s k s with GridVis (overview , g o a l-d ire c te d s e a r c h a n d sorting)

T a sk analysis E xp erim e nta l e va luation

12 p a rticip a n ts - G ridVis v s. S c rolling w indow - within s u ^ e c t s d e s ig n

E xpert ju d g e ratin g s for p a ra g ra p h r e le v a n c e

Ch 6 - A best-case ana lysis

^ ^ e s t - c a s e ana lyses in the abstract

T h e p r o 's a n d c o n 's o l b e s t- c a s e a n a ly s e s

he best case ana lysis of G ridV is ^ S t r a t e g y - le v e l defin ition

S tudy to identify ta g s c e n t s tra te g y (2 p a rtic ip a n ts se le c tin g ta g s with in s tru m e n te d v e rsio n o l GridVis)

P e rform an ce-level definition

G O M S m o d e ls lo r G ridVis u s e

% M odels o l e x p e r ie n c e ■ U n e fitting to d a ta from d o c u m e n ts a n d P a p e r P e r fo r m a n c e E x p erim e n t

T he ben ch m a rk - P aper p e rform ance expe rim en t

24 su b je c ts . G o a W ir e c te d s e a r c h with p a p e r d o c u m e n ts . R etrieval p e rf o rm a n c e a n d p a ra g ra p h re a d in g tim e s m e a s u r e d u s in g v id e o f o o tag e .

M a te r ia ls

D o cu m en t s e le c tio n a n d m anipulation E x p ert ju d g e r a tin g s for p a ra g ra p h r e le v a n c e Q u e ry c o n sfru ctio n

P erform ance m etrics

Ch 7 - R esults fro m th e best-case a n a lysis

/ ^ e b e n chm a rk p e rform ance

N ^ ^ n a l y s i s of high le vel re a d in g str a te g ie s V A nalysis of ta s k p e rfo rm a n c e

G ridV is Vs B e nchm ark perform ance

Figure 1 Thesis Map. The location o f each major

,-,Q)

%

C hapter 2 review s literature from the various areas related to this research. Section 2 on m etadata describes docum ent m etadata from different traditions, looks at som e o f the issues surrounding its production, and review s the w ays it has been used to support search w ithin docum ents. A section on inform ation visualisation, section 3, review s its various techniques and fields evidence to w arrant the design choices in the developm ent o f G ridV is. The follow ing section, section 4, draw s on theories o f inform ation seeking to argue for the potential utility of m etadata taxonom ies, and uses evidence for the im portance o f goal-directed search in the w orkplace to argue that it is w orth supporting and notes the continuous prevalence o f paper docum ent use. The next section, 5, looks at evaluation m ethods. It exam ines the em pirical m ethods and m etrics used to evaluate the support o f goal-directed search w ithin docum ents. A range o f theoretical evaluation

methods are described and the particular choice of techniques used in this research is

justified. One of the more recently developed of these techniques, the use of information

foraging theory, is the subject of the following section, 6, where its foundations, use and

validity are discussed. The final section, section 8, covers visualisation work related to

the work in this thesis, it is critically assessed and the inadequacies of evaluation

techniques which might substitute for the best-case method developed in this research,

are exposed.

Chapter 3 reports two exploratory studies of document use in the head office of a large

UK retailer which attempted to examine the feasibility of using paragraph-level

metadata to support goal-directed search within corporate documents. A subset of the

documents used are identified as being suitable for enrichment with metadata. Their characteristics are described. An analysis of the goals workers bring to documents they

read examines the degree to which these goals can be represented with metadata made

up of keywords or noun phrases.

Chapter 4 describes the design and construction of GridVis, an interactive visualisation

of paragraph-level metadata. The techniques used to construct the high quality metadata needed are described in section 2. These include techniques adapted from previous

research and a novel technique developed in this research, to elicit the information used

to discriminate relevant material. The design of GridVis is described in section 3; the

contribution to the design, of evidence from numerous rounds of evaluation, is

discussed.

Chapter 5 reports a user trial, task analysis and experimental evaluation of GridVis. In

section 1, the description of a user trial of an early static mock-up of GridVis supports

discussion of a number of design problems and highlights methodological concerns. A task analysis of the use of paragraph-level metadata for goal-directed search is then

Chapter 1 section 2 35

reported for use in a review of the GridVis design (see section 2). The final section of

this chapter, section 3, describes the evaluation of a full implementation of an improved

design, using a controlled experimental comparison with a simple scrolling window.

Results from this experiment are used to argue for the need for an evaluation

methodology which can assess the ultimate potential of a class of designs.

Chapter 6 describes the best-case analysis method and its application to the interactive

visualisation of paragraph-level metadata. The best-case analysis method is described in

section 1.1. Its general advantages and limitation are discussed. In section 2 the best-

case model of GridVis use which incorporates information foraging theory is described.

First, in section 2.1, the model is described at a high, or strategy level. Then, in section

2.2, it is described at a computational, or performance level. Section 3 describes an experiment to measure human goal-directed search performance with paper documents.

Chapter 7 reports the results of the best-case analysis. The results from the experiment on performance with paper are reported first in section 1. The navigational strategies

used with paper are described and used to argue the validity of comparing the

performance on paper with that computed by the best-case model of GridVis. The best-

case performance of GridVis is compared with that of paper in section 2. The relative advantages of various affordances offered by GridVis are reported and an assessment is

made of the general viability of interactive visualisations of paragraph-level metadata.

Chapter 8 summaries the contributions and conclusions offered by this research and describes various avenues for further work.