Creating and Using a Lifelog

There is no universal consensus over the processes that pertain to lifelogging, since different lifelogging systems appear to implement different processes or stages for producing and using a lifelog. Nevertheless, the following processes have been identified in literature and are presented here as the most frequently reported:

Capture. The first step is the collection of raw data (e.g., pictures, videos, locations, etc.) that correspond to a life experience or event, commonly with the use of a lifelogging device [93, 184, 209]. Data capture typically occurs automatically, continuously, and transparently, without the user’s intervention. Often, captured data has to be readily available, and hence the employed lifel- ogging devices are continuously streaming captured data to a central storage for real-time analysis. Application scenarios based on readily available data include supporting one’s prospective memory (i.e., memory about the future), or simply logging for security reasons (e.g., an car dash cam). Alternatively, captured data may be opportunistically offloaded to the central storage of a lifelogging system (e.g., when Wi-Fi connection is available). Evidently, the captured data can be immensely heterogeneous (e.g., visual, audible, spatial, physiological, activity type, etc.), since it originates from the collection of multi-modal data streams from a plethora of sources, such as sensors (e.g., portable or fixed cameras, mi- crophones, wearable and environmental sensors, etc.), services (e.g., stored e- mails on a mail server), (mobile) applications (e.g., chat transcripts), and net- works (e.g., social media posts and activity), typically by accessing an Application Programming Interface (API)[93].

Synchronization. The great heterogeneity that characterizes captured data renders the synchronization process as the necessary next step. At this stage, captured data may be re-formatted to match formats employed by the lifelog- ging system, and are temporally and spatially synchronized, particularly when it comes to handling multi-modal data streams [93]. Then, data is filtered and clustered. Ideally, data synchronization is immediate and automatic, but often a delay is entailed due to data offloading employed by lifelogging devices. Syn- chronization can be instantaneous and automatic when the lifelogging device is in constant communication with the central storage.

Segmentation. Once data has been synchronized, (temporally, spatially, etc.), related data chunks are combined and restructured for forming mean- ingful data segments, usually in the form of events or episodes, which can be utilized as discrete "units of retrieval" [93, 184]. Retrieval units, in the form of episodes, should be viewed as single "semantic units" that can be ordered based

37 2.2 Lifelogging

on temporal or spatial criteria [102]. These episodes represent the actual life experience of the user, and are linked to each other in a fashion that facilitates the reconstruction of the actual experience, in a similar way as human memory does (e.g., in temporal order) [231]. These episodes can be utilized in subse- quent analyses for determining their uniqueness and regularity, thus extracting behaviour patterns for supporting additional purposes (e.g., for goal-tracking and self-reflection[210]).

Enrichment. At this stage, episodes are semantically and emotionally en- riched for creating a sort of narrative or story. The enrichment takes place by labelling the episodes with meaningful meta-data, such as occasion, place name, people, activity, feelings, and others, as it is required for enhancing the useful- ness of the episodes that constitute a lifelog[93, 94, 150]. The information for the enrichment is derived from additional sources, such as data from diverse sensors, information from temporally adjacent episodes, APIs, semantic analy- sis, face recognition, and others, so that new meaning can be attributed to an episode. This process can be automatic or manual, with users reviewing and annotating their data, though only a few ever retrospectively edit, manage, or curate their data (e.g., deleting duplicates) [70]. Depending on the application of lifelogging per se, a rich narrative summary could be constructed for support- ing keyword text search. The enrichment process is thus critical from an indexing perspective, since the elicited labels (i.e., meta-data annotations) can be used for retrieval in the next stage.

Retrieval. At this stage, the lifelog related to a life experience has been cre- ated and is ready for retrieval, typically via a search engine [93, 94, 150, 184]. We need to stress that the successful retrieval of a lifelog is heavily depending on the nature of the circumstances under which a lifelog is about to be used. This im- plies that understanding user’s intentions in how lifelogs are accessed and used, in different (lifelogging) application scenarios, is of utmost importance. Based on the lifelogging application scenario per se, a Lifelogging User Interface (LUI) is responsible for providing the appropriate retrieval functionalities and controls. Interaction. A Lifelogging User Interface (LUI) enables the communication between the user and the lifelogs [94, 184]. For doing so, a LUI encompasses the necessary graphical elements for furnishing input and output control, and thus facilitating overall interaction. Typically, a LUI hosts search and retrieval options and functionalities for actuating the lifelog retrieval process. In par- ticular, lifelogs can be retrieved, presented, and visualized, using appropriate modes of presentation, visualization, and information prioritization. For exam- ple, when one interacts with a LUI for supporting one’s memory, the following user interaction pattern is followed: (a) overview, (b) zoom and filter, (c) details

38 2.2 Lifelogging

on demand. Depending on the application of lifelogging per se, the input con- trols of a LUIs usually support assisted query formulation, engaging storytelling, summarization, visualization, and the delivery of information that calls for ac- tion (e.g. recommendations or reminders). The interaction with a LUI can be multi-modal (e.g., on personal computers, tablets, smartphones, smartwatches, ambient displays, etc.) for rendering a lifelog more accessible and pervasive. Ideally, the interaction with a LUI should be the only "touchpoint" that requires the active participation of the user for retrieving a lifelog. However, not all lifel- ogging systems implement the aforementioned lifelogging processes, or even if they do, they may require user intervention. When a lifelogging system supports the prospective memory (i.e., remind one to drink more water), the interaction is usually initiated by the system via a LUI (e.g., display reminder on one’s smart- watch). As such, the lifelog is retrieved and presented seemingly unexpectedly for the user, based on some contextual criteria (e.g., time, location, etc.).