Many systems obtain indicators of item importance from system users, either implic-itly or explicimplic-itly in order to improve the performance of the system, or help locate items or information of importance/relevance to the user. Explicit relevance feedback is where the user themself informs the system of the importance of documents to their interests and needs. This explicit relevance feedback can be obtained from a num-ber of sources [Brusilovsky, 1996], for example through user feedback where the user grades the relevance of items, a good example here is the ’thumbs up’ and thumbs down’ facility offered in many blogging systems for example, or through adaptation of the system at the backend (user model) or interface level, for example systems that allow users to add or remove items from their ’interested in list’. While explicit feed-back can be a good way to discern the interests of users, this benefit is at the cost of the cognitive burden placed on the user [Belkin et al., 2000]. An alternative is implicit relevance feedback, or as we refer to them in this thesis implicit indicators of item importance, whereby the system attempts to infer how interesting the items are to an individual [Balabanovic, 1998, Ruthven, 2005]. Implicit indicators of interest can po-tentially be obtained from actions performed by individuals on items. Examples here include: actions on the web such as printing, saving, forwarding, bookmarking, reply-ing to and postreply-ing a follow-up message to an item can indicate interest; returnreply-ing to
the previous document without having either saved the target document or followed further links can indicate disinterest; and since there is a tendency to browse links in a top-to-bottom, left-to-right manner a link that has been ’passed over’ can be assumed to be less interesting [Lieberman, 1997]. Other examples here include: the number of times a person visits/views an item can indicate their level of interest in the item; and the time a user spends reading an item can also indicate their level of interest in the item [Nichols, 1997]. Much of this research on the use of implicit relevance feedback has been focused on web search, information filtering applications and recommender systems. Here implicit relevance feedback has been used either as a current indicator of item importance to guide a current search (e.g. the ’show me more like this no-tion’), or to form a user profile which provides details on the interests of users and in turn can be used to guide future search tasks (explicit relevance feedback is used in the same way). All of these implicit indicators of item importance, either future or current, are personal to the individual.
A different type of indicator of item importance, which is not personal to the individ-ual, is the use of the web’s link structure to detect the importance of web pages using algorithms such as PageRank [Page et al., 1998] and HITS [Kleinberg, 1999]. These implicit indicators of webpage importance are then used as static query independent boosts in retrieval algorithms. Use of algorithms such as PageRank and HITS has moved beyond the web into the personal file search space. [Chirita and Nejdl, 2006, Chirita et al., 2006, Kurland and Lee, 2006, Kurland, 2006, Soules and Ganger, 2005, Soules, 2006] used varying approaches to link items or result lists in desktop col-lections. These future personal, link based, indicators of item importance, are then used to re-rank the results of ranked retrieval result lists. More specifically, [Chirita and Nejdl, 2006, Chirita et al., 2006] factor link based indicators of item im-portance, derived based on access patterns between files and shared characteris-tics of files (e.g. linking files in the same folder) into the ranked retrieval score;
[Kurland and Lee, 2006, Kurland, 2006] retrieve a set of documents in response to a user query using traditional methods and then use inter-document relationships to rank order the retrieved documents; and [Soules and Ganger, 2005, Soules, 2006] re-ranks the results of text-based queries using link based indicators of item importance derived from past access patterns between files.
Exploration of the many facets of implicit and explicit user feedback and user
mod-elling is beyond the scope of this thesis. In this thesis we are interested in exploring a new type of implicit indicator of future item importance and its potential utility as a static, query independent, score integrated into ranked retrieval approaches for the lifelogging domain, namely biometric response associated with past experience of lifelog items. For the remainder of this section we overview biometric response and its existing uses in the digital environment as an implicit indicator of current item importance.
2.3.1 Biometric Response
As mentioned in Chapter 1.1.2, previous work has shown an individual’s biometric response to be related to their overall arousal levels [Lang, 1995]. Significant or im-portant events tend to raise an individual’s arousal level, causing a measurable bio-metric response [McGaugh, 2003]. Events that can be recalled clearly in the future are often those which were important or emotional in our lives [Gazzaniga et al., 2002].
It has been demonstrated that the strength of the declarative or explicit memory for such emotionally charged events has a biological basis within the brain, specifically involving interaction between the amygdala and the hippocampal memory system [Ferry et al., 1999]. Variations in arousal level elicit physiological responses such as changes in heart rate (HR) or increased sweat production. Thus one way of observing an arousal response is by measuring the skin conductance response (SCR) (also re-ferred to as galvanic skin response (GSR)). The GSR reflects a change in the electrical conductivity of the skin as a result of variation in the activity of the sweat glands. It can be measured even if this change is only subtle and transient, and the individual con-cerned is not obviously sweating [W. Boucsein, 1992, Gazzaniga et al., 2002]. The rate of heat exchange from a person’s body to the outside environment, called heat flux (HF), also provides an indicator of an individual’s arousal levels. Arousal response can also be observed through skin temperature (ST). With increased arousal levels, sympathetic nervous activity increases, resulting in a decrease of blood flow in periph-eral vessels. This blood flow decrease causes a decrease in ST [Kataoka et al., 1998, Sakamoto et al., 2006]. Current technologies enable the capture of a number of bio-metric measures on a continuous basis. For example using a device such as the Body-Media SenseWear Pro II armband6[Andre et al., 2006] which can continuously record
6http://www.bodymedia.com/ (September 2011)
the wearer’s GSR, ST and HF, or using the Polar heart rate monitor7 which can con-tinuously record the wearer’s HR.
A problem for arousal level detection using biometric response is that many fac-tors, such as defective sensors and food intake, can cause noise in biometric data [Jain and Ross, 2004]. Noise in biometric data when attempting to use it to infer arousal levels is also caused by external factors such as physical activity, which also causes changes in biometric levels [Nakayama et al., 1977, Torii et al., 1992]. One way to measure levels of physical activity is through an energy expenditure calculation which considers a person’s motion, age, weight and height. Energy expenditure is a calculation of the energy used by the human body, based on physical activity, resting metabolic rate and the thermic effect of food (cost of processing food for storage and use) [Ainsworth et al., 1993, Ainsworth et al., 2000, Black et al., 1996, Brockway, 1987, Denzer and Young, 2003]. Devices such as the BodyMedia SenseWear Pro 2 armband record, in addition to biometric readings, a person’s acceleration and provide the op-tion to enter one’s weight, age and height. Using this data the BodyMedia device can calculate energy expenditure readings at a rate of once per minute, using propri-etary algorithms which calculate energy expenditure based on the activity of the user, inferred from the on device data [Andre et al., 2006]. The validity of the BodyMedia SenseWear Pro 2 armband’s energy expenditure calculation has been shown in various studies [Cole et al., 2004, Fruin and Rankin, 2004, Jakicic et al., 2004, King et al., 2005, Mealey et al., 2007, St-Onge et al., 2007]. While not, to our knowledge, explored to date, we believe that consideration of energy expenditure levels when attempting to infer arousal levels from biometric data may remove the noise in biometric data caused by some factors.
2.3.2 Biometric Response and the Digital Environment
Much research exists on exploring the relationship between biomet-ric response and individuals’ arousal and emotional levels, for example [Bradley et al., 2001a, Bradley et al., 2001b, Kim et al., 2004, Kim and Andre, 2008a, Kim and Andre, 2008b, Lang et al., 1993, Lang, 1995, Lisetti et al., 2003, Lisetti and Nasoz, 2004, Maltzman and Boyd, 1984]. Researchers have also begun looking at how an individual’s biometric response may be used in emotion
detec-7http://www.polarusa.com/ (September 2011)
tion for HCI systems, for example [Anttonen, 2002, Anttonen and Surakka, 2005, Klein et al., 2002, Partala and Surakka, 2004, Picard, 2000, Picard et al., 2001, Scheirer et al., 2002, Ward et al., 2002, Ward and Marsden, 2003] and in elic-iting of emotional response to movies and movie scenes, for example [Canini et al., 2010, Chen and Segall, 2009, Hettema et al., 2000, Mooney et al., 2006, Rothwell et al., 2006, Smeaton and Rothwell, 2009, Soleymani et al., 2008].
Research has also been carried out looking at the use of observed biometric response to detect tasks or items in different test sets which are of current relevance or im-portance to the individual. To our knowledge, at present there is only one example of work in this domain, that of selection and elicition of topical relevance for imper-sonal multimedia collections (TRECVid [Smeaton et al., 2007] and TREC Web track [Bailey et al., 2003] collections) [Arapakis et al., 2009]. In this work the authors show a relationship between the topical relevance of search results and an individual’s emo-tional response, where emoemo-tional response is detected by passing biometric measures through a Support Vector Machine (SVM). This work represents exciting and promis-ing progress in support of biometric response as an implicit indicator of current item relevance (or importance) for retrieval systems. However, to our knowledge previous research has not investigated the exploitation of observed biometric response as an implicit indicator of future item importance, nor has it looked at personal lifelog col-lections. This we believe is an important previously unexploited opportunity to gain passive feedback from subjects to potentially improve the retrieval performance of fu-ture searches in both lifelogging and other domains. In Chapter 6 we investigate our hypothesis that there is a relationship between biometric response at the time of expe-riencing items and the future importance of the items. Following this, in Chapter 7 we investigate the utility of these biometric response measures in ranking ranked re-trieval result lists by adding the biometric measures as static implicit indicators of item importance to ranked retrieval algorithms. Our studies on the use of biometrics in re-trieval are not comparable with those of Arapakis et al [Arapakis et al., 2009]. Their studies recorded biometric response in a controlled lab environment, whereas ours record biometric response ’in the wild’. Further, they used impersonal data, whereas we are dealing with personal collections; and they are examining the use of observed biometrics in the detection of current importance of items - that is they are attempting to detect, using biometric response, the items which are relevant to a given search,
whereas we are examining the use of observed biometric response as a future indi-cator of item importance - that is we are attempting to determine whether biometric response associated with previous experience of items indicates the importance of items in collections as a whole.