In this chapter we reviewed existing work in personal information retrieval and pro-vided support for the retrieval techniques that we will explore in this thesis. These retrieval techniques will explore the integration of recalled content and context with query independent biometric context in retrieval techniques for the lifelogging do-main. In order to explore such retrieval techniques test collections are required for evaluation purposes. As we saw in this chapter evaluation in the personal domain is challenging. Further PLs generally span a long time period, potentially up to a life-time’s worth of digital data could be contained in a PL. PLs will additionally typically contain many types of digital media, for example, computer activity, mobile phone activity. This means that to reliably evaluate retrieval approaches in the PL domain long term multimodal PLs are required. Further, in order to facilitate the automatic annotation of rich context sources to PL items, both sensor readings (such as GPS to al-low auto detection of geo-location) and individuals’ interactions (accesses) with lifelog items (e.g. to facilitate tagging items with all geo-locations experienced by the indi-vidual when accessing an item) need to be recorded. In the next chapter we describe how we created such collections for our evaluation purposes.
CHAPTER
THREE
Lifelog Test Sets
Chapter Overview: In order to investigate approaches to integrating re-called context and query independent biometric context into retrieval al-gorithms for the PL domain it was necessary to create test datasets for experimentation purposes. This chapter describes the design and contents of the PL test sets created for this study and the techniques used to cre-ate them. We first introduce our lifelog test sets which consist of lifelog items, including computer files interacted with, emails sent and received, webpages viewed, SMS messages sent and received, and SenseCam im-ages captured, and the context data associated with these items (i.e., title;
path to file; URL; extension type; to; and from) and context data associ-ated with each access to these items (i.e., “date-time” relassoci-ated information;
geo-location when accessing the item; light status when accessing the item;
weather conditions when accessing the item; people present when access-ing the item; and biometric response when accessaccess-ing the item). We then describe the database structure used to archive these test sets. This is fol-lowed by technical details of the means used to log subjects activities to populate these lifelog data sets and a detailed analyses of their contents.
3.1 Introduction
PLs generally span a long time period, potentially up to a lifetime’s worth of digi-tal data could be contained in an individual’s PL. They typically contain many types of digital media, for example computer activity, mobile phone activity, digital pho-tographs, etc. This means that in order to begin to explore and seek to evaluate re-trieval approaches in what begins to approximate a PL of realistic size, personal data collections gathered over a substantial period of time are required. With these points in mind, as part of the iCLIPS project1, personal digital data was recorded over an extended period by 3 postgraduate students within our research group (1 male, 2 fe-males; from Asian and Caucasian ethnic groups)2. Specifically PC and laptop activity, SMS messages sent and received, passively captured images depicting their life (using the SenseCam device, described in Chapter 2.2.3), digital photographs taken, and loca-tion and social context (i.e. geo-localoca-tion and co-present Bluetooth devices from which people present can be inferred, described in Section 3.3.4.3) were recorded over a pe-riod of 20 months by the 3 postgraduate students. Biometric data was also recorded for a one month period (September 2008). It was not possible to capture biometric data for a longer time period due to the cumbersome nature of the biometric devices and psychological burden placed on subjects recording biometric data, described in Sec-tion 3.3.4.3. These recorded personal digital data types and the means used to record them are described in detail in Section 3.3.
These personal data collections are larger and richer than any others we know of and while collected for only 3 subjects their long term nature and richness provide us with unparalleled real lifelogs for experimentation in this emerging domain. While these lifelogs, to our knowledge, are unrivaled, it should be noted that this is not a commer-cial system, rather a research driven approach to creating lifelogs for experimentation purposes. The lifelogs consist of real data, gathered from real sensors and are
sub-1http://www.cdvp.dcu.ie/iCLIPS/ (September 2011).
2One of these test subjects was the thesis author. The other two subjects also conducted their own research experiments using the generated lifelogs. In the investigations presented in this thesis, the author was not an outlier in experimentation. Subjects, the author included, were provided instructions for generation of queries on their personal collections (as will be described in Chapter 4). The author, as a subject in the study, followed the same guidelines as the other two subjects. Subjects also rated the relevance of retrieved results for their user queries, using a simple Boolean relevant/irrelevant rating scale (also described in Chapter 4), which didn’t afford opportunity for removal of objectivity on the author’s, as subject, part. As will be described in Chapter 5, subjects were also required to rate SenseCam images and computer files, with varying associated biometric response, on a number of scales. Subjects, the author included, were not aware of the biometric response associated with images and files, hence objectivity was not removed by the use of the author in this study.
ject to failures of hardware, software, input and output at various points in the data processing chain [Byrne et al., 2010]. This is reflective of a real environment, although hopefully a commercial system would be more reliable. In our personal data gath-ering we are pushing the limits of the available data collection devices and software.
Further, this is real data from individuals, as such it is subject to the individual’s oc-casional need for privacy, need for mental breaks from the lifelogging process, and subjects’ forgetfulness in turning several lifelogging devices on, charging devices and in downloading data from mobile devices before device memory fills (this forgetful-ness results in inability to continue data recording until a download is made and in extreme cases corruption of data onboard the device) [Byrne et al., 2010]. These issues and the resulting implications on the make up of the personal data collections are discussed in Sections 3.3 and 3.4.
This thesis centers on the development of IR algorithms to enable individuals to search for items in their PL collections. We investigate the utility of using a subject’s recalled content and context associated with required PL items for lifelog item retrieval and investigate development of retrieval algorithms designed to exploit these recalled fea-tures. Development of ranked retrieval techniques for the many types of media which may be present in a lifelog, e.g., audio, images, textual items, etc, is beyond the scope of one thesis. In this thesis we focus on the development of ranked retrieval algorithms for the textual media within PLs. Possibilities for extension to other types of media are discussed in the context of future work in Chapter 8.2.2. For the textual ranked retrieval experiments presented in this thesis the 20 months of PC and laptop activ-ity and SMS messages sent and received were organised into lifelog test sets. These test sets contain the content of items (i.e., content of SMS messages and content of the computer files, webpages and emails interacted with) annotated with rich sources of context data derived using the location and social context and information contained within the items. Our choice of context data types was motivated by existing memory studies investigating individuals recall of context associated with items, discussed in Chapter 2.2. Specifically, each item was annotated with the following context types:
words in item title (for computer files title = filename, for emails title = email subject field, for webpages title = title of webpage); extension type; path to file; URL (for web-pages only); and to/from (for SMS messages and emails only). Each access to these items was also annotated with the following additional sources of context data: year;
season; month; day of week; weekday or weekend; beginning of week, mid-week or end week; part of day (i.e., morning, afternoon, evening and night); begin date and time; end date and time; device (e.g., laptop, mobile phone); light status (i.e. daylight and dark); weather; geo-location; and people present. Table 3.1 provides the complete list of context types used. The derivation of context types and organising of recorded personal digital data into lifelog test sets is described in Sections 3.2 and 3.3.
General Information
Item ID Item content
Item Specific Context Data
Title Extension Type
i.e., computer filename, email subject, webpage title e.g., Excel, Web
Path to File URL
To From
(for emails & SMS messages only) (for emails & SMS messages only) Context Data Assigned to Each Access to an Item
Begin Date & Time End Date & Time
Year Season
Month Day of Week
Weekday or Weekend Part of Week
i.e., begin week, midweek, end week
Part of Day device
e.g., morning, afternoon e.g., PC, mobile phone
Geo-Location Light Status
i.e., daylight, dark
Weather People Present
e.g., raining, cloudy e.g. Joe Smith
Table 3.1: Complete set of content and context.
This thesis also explores the possibility of improving retrieval effectiveness using bio-metric bio-metrics captured (for one month) in the context of textual items in a PL and methods by which these might be integrated into our IR algorithms. As part of our investigation into improving ranked retrieval effectiveness for the textual items in PLs using biometric metrics, we explore the utility of captured biometric metrics in detect-ing important lifelog items. Usdetect-ing biometric significance measures to locate impor-tant items from amongst the possibly vast number of items within PL collections also has direct utility in its own right. For example, in the suggestion of interesting items when browsing a lifelog collection. Hence, given the possibly vast number of Sense-Cam images in a lifelog and difficulty in locating interesting images from within such a collection, described in Chapter 2.2.3, in our investigation of important lifelog item detection using biometrics we also consider, in addition to textual media, SenseCam images as an example of the utility of our approach for other lifelog media types. To
facilitate experimentation into extracting important items from lifelogs using biomet-ric response and into re-ranking ranked retrieval result lists using biometbiomet-ric data, one month of the generated 20 month lifelog test set was further annotated with biometric data (described in Sections 3.2 and 3.3). Specifically GSR, HF, ST, HR and energy ex-penditure (described in Chapter 2.3.1) levels associated with accesses to lifelog items.
SenseCam images for this one month period were also added to the lifelog test set (described in Sections 3.2 and 3.3) to examine important SenseCam event extraction using biometrics.
Subjects’ lifelog test sets were stored locally on their PCs in an SQL database. The structure of these lifelog databases is described in detail in Section 3.2. Following the description of subjects’ lifelog databases, in Section 3.3 we describe how we logged, derived and wrote subjects computer activity (Section 3.3.1), SMS messages (Section 3.3.2), SenseCam images (Section 3.3.3) and context data (Section 3.3.4) to the lifelog databases. We then provide an analysis of each subjects resulting lifelog database in Section 3.4. Finally, we conclude the chapter with pointers to the use of the test sets in the remainder of the thesis.