Preliminary experiments exploring the utility of biometric response associated with past experience of lifelog items in detecting the importance of lifelog items to indi-viduals were conducted on an earlier version of the test sets used in this chapter [Kelly and Jones, 2009, Kelly and Jones, 2010b]. These experiments showed a relation-ship between biometric response and the importance of lifelog events to individuals (albeit 1 and 9 months after the lifelog data had been captured as opposed to the 22 month time interval explored in this chapter). However, these experiments suffered a number of shortcomings in the experimental procedure adopted. In particular, the biometric data from the beginning and end of each period of wearing the biometric devices which was skewed (as discussed in Chapter 3.3.4.4) was not removed from the dataset. Also, energy expenditure values were not calculated on the given
sub-jects personal details (i.e., age, weight and height), due to the devices not being set with the subjects’ personal details (as discussed in Chapter 2.3.1 these personal details are used in the energy expenditure calculation). The experiments presented in this chapter addressed these problems.
Further, a number of lessons were learned from these earlier experiments and subse-quently the experiments presented in this chapter differ in a number of key ways:
1. In this chapter’s experiments for SenseCam events a 20 second window was allowed each side of the events;
2. An additional form of biometric data, heat flux, was considered in this chapter’s experiments;
3. The earlier experiments showed failure to capture textual content for computer items to negatively impact on a subject’s ability to recall items in some instances, hence items missing textual content were not considered in the experiments pre-sented in this chapter;
4. Our prior experiments presented subjects with temporal groups of computer items (events) to rate, however many of the items in such events were unrelated (e.g. viewing email and then returning to a coding task). Subjects’ rating such events does not provide an indication of the role of biometric response in de-tecting the future importance of individual computer items, hence in the study presented in this chapter subjects were presented with single items to rate;
5. In our prior experiments the only way physical activity and other external fac-tors were accounted for was through the deletion of biometric data captured during periods of high energy expenditure (i.e. an earlier version of the re-moveEng technique). Similar to use of the rere-moveEng technique in this chapter, the only computer and SenseCam events which were available for presentation to the subjects’ were the ones which occurred at the times of the remaining bio-metric data captured. In using this approach we removed the possibility for suggestion of potentially interesting computer and SenseCam events which oc-curred during the periods of deleted biometric levels. As highlighted earlier, it is also possible in using this technique that the remaining biometric levels were still influenced by external factors. Hence in the experiments presented in this
chapter, we also explored factoring energy expenditure levels into the biometric readings (i.e. the divEng technique) as opposed to just looking at the removal of periods of high energy expenditure.
While these factors would have negatively impacted on the experiments in the prior work, the experiments nevertheless provided an insight into the utility of biomet-ric response in locating events from lifelogs which individuals may be interested in viewing. The experiments also provide us with some insight into how the SenseCam images individuals wish to view change over time. This was particularly apparent for Subject 3, who soon after capturing SenseCam images and while still actively en-gaged in SenseCam image capture, did not wish to view a lot of mundane events such as cooking dinner captured by the SenseCam. However over time this changed and while no longer lifelogging and capturing such repetitive life events, the subject found pleasure in viewing these ‘mundane’ events (as discussed earlier in this chapter). We can only speculate as to whether further in the future the same will hold true for the other subjects. However we feel that individual difference will come into play both from the point of view of what will be interesting to view and how often subjects will consult their lifelogs. Future work should consider use of qualitative user studies to establish the different patterns of lifelog browsing exhibited by individuals. Lifelog viewing patterns will also, we believe, depend on personal circumstances at given moments in time and on the different stages in one’s life.
From the findings of our studies we speculate that biometrics may prove to be a more useful tool for occasional lifelog browsers, as opposed to those who appear to browse collections on a more regular basis. We also believe that biometrics on their own are by no means the full solution, however integrated into an application which supports lifelog browsing using category facilities, e.g., show me images on topic x, images in y location, images with z person in them, it could prove a useful tool. Further investigation on this topic appears to be justified for future work.
6.5 Conclusions
In this chapter we set out to explore the role of biometric response in detecting impor-tant items within lifelogs. We investigated whether items coincident with maximum
observed biometric galvanic skin response (GSR), heat flux (HF) and heart rate (HR) and with minimum observed skin temperature (ST) readings were more important to subjects, and whether this would mean they would be most useful or interesting for subjects to view in the future. From this study, relationship between biometric levels and both SenseCam event and computer item importance was observed. The Sense-Cam event selection results are important since ability to extract interesting events from vast SenseCam collections is challenging but important, if these archives are to have long-term use. As mentioned previously, while these results are promising, it is acknowledged that this study was conducted on a limited number of subjects over a short period of time. Investigation using more participants over a longer timeframe is required to further test our suggested conclusions.
Overall we saw that HF levels with periods of high energy expenditure removed (i.e., removeEng technique) is most beneficial for detecting SenseCam events subjects may wish to view in the future, but that consideration of other additional factors beyond biometric response may improve performance. ST proved most beneficial for detect-ing computer items individuals may wish to view in the future usdetect-ing the divEng tech-nique.
However, different pictures began to emerge when we looked at the results of individ-ual subjects. HF levels using the removeEng technique was the only one which showed relationship with subjects’ desire to view SenseCam events in the future across the 3 subjects. Indeed, for Subject 3 who regularly browsed their SenseCam collection, no other relationship between biometric response (using either the removeEng or divEng technique) and the subject’s desire to view SenseCam events in the future was ob-served. However, for Subjects 1 and 2 who rarely if ever browsed their SenseCam col-lections, the divEng technique, showed greatest utility in detecting SenseCam events these subjects might want to view in the future, with ST levels showing the greatest utility in this regard. For computer items, while ST levels using the divEng technique proved to be the most useful technique in detecting items Subjects 1 and 3 might want to view in the future, this technique did not prove useful for Subject 2. For Subject 2 GSR level using the divEng technique was the only technique which showed relation-ship with items that the subject might wish to view in the future.
Overall from these results we can suggest that factoring of energy expenditure into the individual biometric readings (through the use of the divEng technique) is
impor-tant. However, it is hard to draw conclusions on a biometric measure which may prove most useful in detecting events/items subjects may wish to view in the future.
Rather, we conclude that biometric levels associated with past experience of lifelog items seem to show promise in detecting important items/events in individuals per-sonal digital collections. Future work needs to be done to explore the nature of bio-metric response associated with lifelog items and the role of this response in detecting important lifelog items in greater detail. This caveat notwithstanding, the results ob-served in this chapter support investigating the use of the biometric tags that can be assigned to lifelog items as static scores for ranked retrieval of personal items in a lifelog. The next chapter combines the work of this chapter with the techniques for ranked retrieval from lifelogs investigated in Chapter 5.
Subject 1 Subject 2 Subject 3
Retrieve Other Retrieve Other Retrieve Other
in Future Questions in Future Questions in Future Questions divEng
Table 6.3: Summary of the relationship observed between subjects’ ratings for Sense-Cam events and biometric levels. Best performing biometric measure for each subject highlighted in bold font, ’<’ refers to little relationship.
Subject 1 Subject 2 Subject 3
Retrieve Other Retrieve Other Retrieve Other
in Future Questions in Future Questions in Future Questions divEng
Table 6.4: Summary of the relationship observed between subjects’ ratings for com-puter items and biometric levels. Best performing biometric measure for each subject highlighted in bold font, ’<’ refers to little relationship.
Figure 6.5: Questionnaire results - average SenseCam event ratings for the three sub-jects combined for max, average (ave) and min GSR, HR, HF and ST using the re-moveEng technique. Note the graphs on the left column present a break down of the results, while the right column graphs present a grouping of the questions 5-point scale.
Figure 6.6: Questionnaire results - average SenseCam event ratings for the three sub-jects combined for max, average (ave) and min GSR, HR, HF and ST using the divEng technique. Note the graphs on the left column present a break down of the results, while the right column graphs present a grouping of the questions 5-point scale.
Figure 6.7: Questionnaire results - average computer item ratings for the three subjects combined for max, average (ave) and min GSR, HR, HF and ST using the removeEng technique. Note the graphs on the left column present a break down of the results, while the right column graphs present a grouping of the questions 5-point scale.
Figure 6.8: Questionnaire results - average computer item ratings for the three sub-jects combined for max, average (ave) and min GSR, HR, HF and ST using the divEng technique. Note the graphs on the left column present a break down of the results, while the right column graphs present a grouping of the questions 5-point scale.
Figure 6.9: Questionnaire results - SenseCam event ratings for Subject 1 for max, av-erage (ave) and min GSR, HR, HF and ST using the removeEng technique.
Figure 6.10: Questionnaire results - SenseCam event ratings for Subject 1 for max, average (ave) and min GSR, HR, HF and ST using the divEng technique.
Figure 6.11: Questionnaire results - SenseCam event ratings for Subject 2 for max, average (ave) and min GSR, HR, HF and ST using the removeEng technique.
Figure 6.12: Questionnaire results - SenseCam event ratings for Subject 2 for max, average (ave) and min GSR, HR, HF and ST using the divEng technique.
Figure 6.13: Questionnaire results - SenseCam event ratings for Subject 3 for max, average (ave) and min GSR, HR, HF and ST using the removeEng technique.
Figure 6.14: Questionnaire results - SenseCam event ratings for Subject 3 for max, average (ave) and min GSR, HR, HF and ST using the divEng technique.
Figure 6.15: Questionnaire results - computer item ratings for Subject 1 for max, aver-age (ave) and min GSR, HR, HF and ST using the removeEng technique.
Figure 6.16: Questionnaire results - computer item ratings for Subject 1 for max, aver-age (ave) and min GSR, HR, HF and ST using the divEng technique.
Figure 6.17: Questionnaire results - computer item ratings for Subject 2 for max, aver-age (ave) and min GSR, HR, HF and ST using the removeEng technique.
Figure 6.18: Questionnaire results - computer item ratings for Subject 2 for max, aver-age (ave) and min GSR, HR, HF and ST using the divEng technique.
Figure 6.19: Questionnaire results - computer item ratings for Subject 3 for max, aver-age (ave) and min GSR, HR, HF and ST using the removeEng technique.
Figure 6.20: Questionnaire results - computer item ratings for Subject 3 for max, aver-age (ave) and min GSR, HR, HF and ST using the divEng technique.
CHAPTER
SEVEN
Static Scores: Boosting Relevant Items in Result Lists using Past Biometric Response
Chapter Overview: Given the relationship observed in the previous chapter between biometric response at the time of experiencing lifelog items/events and the future importance of items to the individual, we wished to explore our hypothesis that use of biometric response as a static query factor boost in ranked content+context retrieval algorithms in the lifelogging domain would prove useful. In this chapter we investigate this hypothesis. Following the chapter introduction the setup of this investiga-tion is described. We then provide a detailed analysis of the results of using Galvanic Skin Response, Heat Flux, Skin Temperature and Heart Rate as static scores in content+context-based retrieval algorithms for the lifelog-ging domain. This is followed by a discussion of the topic of this chapter, and concluding remarks.