approach lies in the exploration and evaluation of novel contextual information for the generation of effective memory cues that are displayed throughout one’s day for augmenting one’s memory recall in diverse contexts. This yields an un- precedented opportunity for eliciting insights that will inform the design and de- velopment of future pervasive memory augmentation systems, implementing an end-to-end (from capture to presentation), customized memory augmentation, that will be delivered in situ, unobtrusively, and continuously via ubiquitous tech- nologies. We expect that the memory augmentation effect will persist even when the system seizes to provide further memory support, greatly reducing the rate at which (episodic and semantic) memories are forgotten (see Figure 3.2).
3.3
Apparatus
In this work, we use a diverse set of devices that can be grouped by purpose in 3 broad categories: data collection, intervention delivery, and feedback acquisition. The data collection category includes devices that capture data that is used in the selection and generation of memory cues. The intervention delivery category includes devices that deliver memory cues (e.g., a smartphone), whereas the feedback acquisition category involves devices purposed for obtaining data used for evaluation purposes (e.g., how effective a memory intervention was). Some devices may serve more than one role simultaneously. In general, we use the following devices:
• Smartphones are the main instruments for delivering our memory inter-
ventionin the form of presenting memory cues. Alternative options would be presenting memory cues on domestic or even public ambient displays, smart watches or VR/AR headsets (see next). A smartphone is typically always available and within one’s reach. Thus, we can assume a maximi- sation of our memory intervention by capitalizing on increased user expo- sure to memory cues displayed on one’s smartphone. A smartphone can also have a feedback acquisition role for collecting in situ feedback regard- ing the memory intervention tested each time, participants’ well-being, af- fective state and others, typically via Experience Sampling Method (ESM) [45], Likert scales (e.g., System Usability Scale [32]) and questionnaires. Furthermore, a smartphone can also be used for data collection that com- plements the process of delivering memory cues. For example, continuous location monitoring can be used for displaying different memory cues when one is at home and when at the workplace. We usually deploy our appli- cation prototypes on participants’ personal devices, hence drawing on the
56 3.3 Apparatus
familiarity they have with their own devices. When certain device criteria are not met (e.g., screen size, OS version, etc.) by one’s personal mobile device, we provide one with an eligible device available in our lab.
Figure 3.3. Left: Narrative Clip 2, Right: Empatica E4.
• Narrative Clip1 is a compact, modern and wearable lifelogging camera that enables the collection of images automatically and manually, by simply double-tapping it (see Figure 3.3). One can simply attach it to one’s clothes and it would capture pictures throughout one’s experience (e.g., a visit in a museum). The Narrative Clip can be considered the descendant of Mi- crosoft’s SenseCam that eventually made the concept of lifelogging known and somewhat popular. The newer version (Narrative Clip 2), comes with an improved resolution, increased storage capacity and an accompanying mobile application that allows one to customize picture capture interval time, while supporting the capture of short (2 min) video segments. We use this device for collecting pictures for supplying our memory cue selec- tion and generation process. Hence, the Narrative Clip serves as a data
collectiontool.
• GoPro HERO42 is a small portable camera (41×59×30 mm) equipped with a wide-angle lens that weighs 83 grams (147 grams with case), and can capture up to 4K resolution videos (3840×2160) and 12 MP pictures. 1http://getnarrative.com/
57 3.3 Apparatus
Figure 3.4. GoPro HERO4 with an external directional microphone, a tripod, and a Wi-Fi remote controller.
HERO4 supports Bluetooth and Wi-Fi connectivity, and comes with a Wi- Fi remote that can pause and resume recording. HERO4 is particularly popular for capturing 1st-person content during (extreme) sports or other outdoor encounters (e.g., time-lapse video), as it is notoriously durable and water-resistant up to 40 m (with case). We utilized HERO4 primarily for its easy-to-deploy nature and for the high quality of recorded video. For improving the quality of recorded audio, we have used a directional microphone that reduces noises from other sources, while for improving the angle of capture, we have used a small tripod (see Figure 3.4). We typ- ically deploy HERO4 for recording participants utterances during a follow- up interview, but also for collecting video that would be later utilized for generating memory cues (e.g., Chapter 7). Therefore, HERO4 serves as both a data collection and feedback acquisition tool.
58 3.3 Apparatus
• Empatica E43is a wristband equipped with a photoplethysmography (PPG) sensor for collecting physiological feedback of close to clinical standards accuracy, while remaining unobtrusive to the engagement one has with an experience (see Figure 3.3). It monitors a series of physiological responses such as Electrodermal Activity (EDA/GSR), activity levels (with 3-axis ac- celerometers), body temperature, Heart Rate (HR), and Inter-Beat Interval (IBI) as a measure of Heart Rate Variability (HRV). Physiological responses can be used for estimating one’s arousal and partially one’s affective state in situ, and is thus a useful indicator of one’s engagement in an experience. Due to indications that physiological responses help us refine our memory cues (e.g., help filter out irrelevant information or indicate the effect of a memory cue), we consider the E4 wristband both a feedback acquisition and data collection tool.
• Smart watches (and fitness trackers) are one of the latest "consumer trends" in the field of wearable computing. Smart watches are bigger than smart wristbands (e.g., Fitbit4 and Jawbone5) because they are also equipped
with a touch screen at the size and shape of typical wristwatches. Smart watches can host a large variety of sensors, some of which are dedicated for monitoring physiological responses. For example, an optical sensor (i.e., PPG) embedded in the back-side that touches the inner part of one’s wrist can detect one’s current heart rate. Though yet not able of achieving the ac- curacy of professional medical instruments, or that of E4 wristband, it can however provide a clue of one’s current physiological state (e.g., arousal levels). Thus, smart watches and fitness trackers can be used for both feed- back acquisition and data collection purposes. Furthermore, a smart watch is designed to function as the extension of smartphones, delivering all types of notifications (e.g., SMS, calls, e-mails, etc.). Despite being equipped with a small screen, we believe that the smart watch holds a potential for displaying memory cues, and hence perhaps able to deliver a mem- ory augmentation (e.g., display a grocery icon for reminding one to do the groceries). We thus consider the smart watch also capable of delivering a
memory intervention, playing the same (triple) role as a smartphone. • Tobii TX3006 stationary eye tracker is the primary instrument we use in
this work for analysing how one’s eye gaze is changing when viewing differ- ent memory cues on novel (mobile) Lifelogging User Interfaces (LUIs). For 3https://store.empatica.com/products/e4-wristband?variant=945527715/
4https://www.fitbit.com/eu/home/ 5https://jawbone.com/