From Lifelogging to Digital Possessions

Augmenting memory through Lifelogging

Computer scientists have been inspired by the possibilities of ‘technologies of memory’ at least since Vannevar Bush's vision of the Memex in his influential essay ‘As We May Think’ (1945). The Memex imagined infinite storage of, and rapid access to, the world’s knowledge; Bush described the hypothetical machine as “an enlarged intimate

supplement to his memory”. This vision foreshadows the archive metaphor’s

longstanding influence in the development of technologies to augment human memory. The Memex vision has been most faithfully enacted on a personal level as part of Microsoft’s ‘MyLifeBits’ project (Gemmell et al., 2002, 2006), with researcher Gordon Bell attempting the ‘total capture’ of his life through ‘lifelogging’. A singularly dedicated pursuit, Bell relied on an extensive array of wearable technologies, in particular the SenseCam (Hodges et al., 2006) an automatic wearable camera that passively and continually takes digital photos. Before being later (unsuccessfully) commercialised (autographer.com), SenseCam inspired much memory-related technology research, with even entire conferences devoted to the novel device (Kerr et al., 2013). Lifelogging envisaged technological solutions to the perceived problem of a fallible human memory. It was not only a case of reminding oneself of important tasks, but appeared to promise a kind of superhuman veridical recall of one’s life. Arguably an elder cousin of the

Quantified Self movement, lifelogging also offered a means to ‘know thyself’ and reflect. As storage space became a more trivial problem, the significant research challenges were focused on designing for effortless capture of more and better data about one’s life, especially through wearable devices and interfaces to search and manage the vast personal records that lifelogging would produce. A 2006 Special Issue on Personal Information Management (or ‘PIM’) is illuminative of the ‘total capture’ philosophy underlying some of this early work. The tagline reflects the pioneering technological spirit: “Cheap and fast search and storage technologies help bring order to our messy personal information environments, freeing us to make the most of our information collections.” (Teevan et al., 2006, p.40).

Beyond Total Capture

HCI researchers have since questioned and sought to refine the lifelogging vision with a more user-centred approach. Specifically, Sellen and Whittaker (2010) called for the field to look ‘beyond total capture’. Their constructive critique described the need for “a more precise specification of what it means to support human memory” (p.72). They underline the need for a psychological basis to strategically target weaknesses in human memory, and to design systems that are more situation specific – for example, supporting memory in a classroom or meeting. To this end, they offer ‘Five Rs’ for types of remembering which HCI researchers might support: recollecting, reminiscing, retrieving, reflecting and

remembering intentions. Whittaker’s own work (Whittaker et al., 2012) takes this further and proposes ‘synergetic recollection’, emphasising that many of the unique faculties of remembering cannot easily be delegated to technology. Similarly, Bannon (2006) was one of the first HCI researchers to suggest ‘forgetting as a feature’ necessary for lifelogging systems.

From a PIM perspective, Cathy Marshall’s work (Marshall et al., 2006; Marshall, 2007) sought to approach the design of digital archives from studying people’s current practices of managing material archives. She noted how people’s strategies of ‘benign neglect’, which generally served them well enough for physical things, were poorly adapted to the management and accumulation of ‘digital belongings’, especially over the long term. Marshall et al. (2006) paint a grave picture of how personal digital possessions are insufficiently safeguarded and are at risk from an approach to “save everything now, decode it all later”. Instead, there is overwhelming “digital dross” caught in the net; curatorial efforts that apply to institutions rarely translate into the home. Similarly, Kaye et al. (2006) studied the material archiving practices of academics, asking not only how people manage, but more crucially, why do people archive at all? This work questions the inherent value of vast archives, and proposes a refinement of ‘total capture’ towards the present needs and actions that such archives actually serve.

Speaking to lifelogging directly, Petrelli et al. (2009) undertook a study that asked families to create a time-capsule together. Though simple, this work underlined the diversity of motivations and material in creating such an archive. What families chose to ‘capture’ in their time-capsules were emphatically not materials to support veridical recall, but ways to stimulate reconstructive remembering in the future, from a diversity of memory cues; photographs, children’s drawings, newspaper cuttings, small toys, etc. Drawing on Peirce’s ‘typology of signs’ (see Atkin, 2013) they categorise the objects in the time capsule as an icon, index, or symbol. Icons resemble or imitate the

corresponding object to be remembered; a photograph of the family home. Indexes relate physically or causally; home recipes index a family’s passion for cooking. Symbols only relate to the object by those who can recognise and interpret it. For example, a knife and fork “represent eating together as a cultural statement”. Such semiotic analyses are a further perspective on mediation and suggest distinct intentions and practices for remembering different aspects of one’s past.

Together, studies such as these have helped to refine subsequent memory research in HCI, and to move away from a purely archival perspective or a quest for veridical recall. van den Hoven and her colleagues have led a program of work focused on ‘memory cues’ (e.g. Hoven and Eggen, 2007; Mols et al., 2014; van den Hoven and Eggen, 2014).

Drawing particularly on Autobiographical Memory (AM) theory (Cohen and Conway, 2007), their work approaches the design of external memory cues (from objects to people and places), seeking to design the form, time and place of such cues to better support personal remembering in everyday life. Also on a psychological footing, significant work has focused on technology to support reminiscence and reflection often positioned as a personal and societal good (e.g. Cosley et al., 2012; Fleck and Fitzpatrick, 2010; Frohlich and Murphy, 2000; Peesapati et al., 2010; Thiry et al., 2013). Peesapati et al. (2010) designed the ‘Penseive’, a system that “supports everyday reminiscence by emailing memory triggers to people”. These triggers consisted of varied fragments of past social media content, or prompts about common life experiences, which participants were invited to respond to. Across all of these studies, reminiscence and reflection are actively encouraged as positive behaviours (following Bryant et al., 2005) and are often imagined as specific activities, involving setting time aside to do remembering. However, HCI research has increasingly sought to understand how technologies of memory are embedded in remembering as an ongoing and mundane lived experience.

Memory-as-a-resource-for-action

However, an everyday turn in HCI memory research is perhaps best expressed by Harper et al. (2008), who echo the work of Bartlett (1932) and Middleton and Brown (2005) in considering ‘memory-as-a-resource-for-action’. Their approach applies prior debates about memory-in-the-head and memory-in-the-world, to reimagine the role and possibilities of lifelogging devices like SenseCam. With an ethnomethodological sensibility focused on action, they ask what present aims and actions remembering achieves, when looking back at chosen SenseCam images from thousands captured during a brief time spent wearing the device. Their study shows that remembering is much more than simply a question of achieving efficient and veridical recall of the past. Remembering is an ongoing act oriented towards the demands of the present; it involves the imaginative construction and reconstruction of specific personal narratives. In its undertaking, meaning is made through, for example, constructing one’s identity, forging and maintaining social bonds, or connecting to a place.

Harper et al.’s approach (2008), echoed in much subsequent work, questions the

fundamental assumptions of lifelogging, which assumes that more or ‘better’ memory is an inherent good. Instead, their focus turns to how people remember in the course of everyday life or in Bartlett’s terms once again – their ‘main preoccupations’. Research inquiries based on such thinking investigate the existing and emerging role of technology in remembering more broadly; rather than applying technology as a salve to a problematic human memory. This perspective sees remembering as situated action (Suchman, 1987), happening ‘on the fly’ and contingent on local settings. As we shall see, this invites more holistic discussion of remembering as it relates to specific contexts, practices and values, and admits broader design perspectives around, for example, narrative or temporality. This rather ‘third wave’ HCI perspective (Bødker, 2006) represents a refinement of the lifelogging vision not only on a psychological basis (as achieved by Sellen and Whittaker, 2010), but a situated, socio-cultural and experienced-centred one.

Digital Possessions

The SenseCam studied by Harper et al. (2008) is an explicit lifelogging technology. However, the ‘new memory ecology’ described by Brown and Hoskins (2010) suggests that as almost all of our digital interactions are recorded, we are surrounded by

technologies of memory in one form or another. Intentionally or otherwise, these produce a vast archive which includes for example: a myriad of personal and work-related

documents and files; digital media collections (e.g. photos, videos, music); online accounts and avatars (particularly social media profiles); thousands of emails, message archives and metadata containing histories of online interactions; and location history. Collectively, these personal materials have become characterised as digital or virtual possessions (Lindley et al., 2013; Odom et al., 2011, 2014a). This framing extends earlier PIM research; shifting focus from efficient access and management, to how people orient to and experience their burgeoning digital archives. A focus on possession aptly captures the extent to which such an archive increasingly reflects one’s identity. This invites broader considerations of material culture, and attention to the potentially distinctive features of digital possession versus the well-understood roles and possession of physical things.

Importantly, Odom et al. (2012) suggest that the very concept of possession or ownership of digital things may be “lost in translation”; especially as digital things stretch beyond

individual devices or networks, and are maintained via the cloud. This picture is complicated further when we consider not only personally crafted objects, like digital photographs and files, but also the steady accrual of digital traces and data.

Many researchers in HCI understand the nature of possessions through the work of Russell Belk, a researcher in consumer behaviour. (Belk, 1988) offers a synthesising framework of possession, relating William James’ (1890) conceptions of the self, with a long history of anthropological research. Through ‘self-reflection’ on and ‘self-

presentation’ of their things, possessions become valued, to form an ‘extended self’. People use their possessions to develop “a sense of who they are and who they wish to be seen as” (Odom et al., 2014).

Building on Belk’s theory, Odom et al. (2012, p.790) emphasise “that to possess is not merely a noun nor a verb, but a complex set of actions that transform the relationship between a thing (virtual or physical) and a person.” Understanding these actions has been the focus of many inquiries into people’s practices with physical and digital things, especially at home or between families, and as it relates to crafting a digital legacy (e.g., Banks, 2011; Brubaker et al., 2014; Durrant et al., 2009b; Golsteijn et al., 2012; Gulotta et al., 2014; Kirk et al., 2010; Kirk and Sellen, 2010; Moncur et al., 2015; Petrelli and Whittaker, 2010a). This body of HCI scholarship almost universally emphasises value- centred design perspectives that propose designing interactions commensurate with existing human practices and values regarding possession.

This value-centred perspective is quite distinct from the earliest computer scientists who envisioned lifelogging technologies to support the total capture of one’s life. The coherent archive they imagined, would augment and even supersede human memory. However, rather than being used to achieve efficient veridical recall, as imagined through the Memex, digital possessions are implicated in much more complex human experiences and relationships. These possessions are produced, or at least preserved, by an array of technologies of memory. The diversity and ambiguity of what constitutes these digital possessions, and how they can possibly be accounted for and called upon appears

overwhelming, in contrast, for example, to the more well-defined socio-cultural roles of a family photo album. This thesis argues for quantified data (which is just as wound up in identity and culture) as yet another kind of digital possession to reckon with.

But whatever their particular material configuration, these possessions are a resource for all kinds of present action. The following section pursues this thread, to unpick the values and practices of remembering achieved through interaction with digital possessions.

2.3.3. Remembering with Digital Possessions

As I have indicated, ‘digital possessions’ now traverses a vast literature. What I intend to do here is to surface the values and experiences of remembering with digital possessions which have particular resonance for this thesis.

Defining the Self and Social Relationships

As Belk’s work suggests, possessions are commonly understood to be valued as a means of defining the self: introspectively, and to others. Petrelli et al. (2008), and Kirk and Sellen (2010) home studies evidence people physically surrounding themselves with objects, which embody elements of their past, and support remembering. But by delving into home archives multiple different identities are shown to be at play: of individual members, of parents, of the wider family etc. In fact, possessions can be especially valued because they make often intangible social connections visible; they can index a shared history, or symbolise shared values. Such artefacts provide a unifying perspective, and reflect Middleton and Brown’s (2005) description of a collective framework, which orients how remembering, collectively, can take place.

However, contesting historic visions of a unitary or 'all-defining personal digital archive (e.g., Bush, 1945; Stevens et al., 2003) suggest that, at least online, “a single archive cannot represent different facets of the self” (Lindley et al., 2013) . Participants in their study of web archiving seemed clear that things online also had their proper place; to collapse the separation between baking interests on ‘Pinterest’, a gaming profile and one’s Facebook page would undermine their work in making and curating those distinct and individual online spaces.

Records of emails, messaging and social media use emphasise the extent to which others are always implicated in one’s personal archives. Facebook has employed an explicit ‘See Friendship’ feature, which allows friends to see their interactions with each other on the site. Schwanda Sosik et al. (2012) studied friends’ reflections with the tool, and found humorous media content in particular could signal closeness and reinforce the reasons for

their friendship. However, many found the representation was incomplete, or even potentially misleading, where Facebook activity was ‘superficial’ or under-represented negative events. This research highlights the limits to any digital traces on their own adequately or holistically representing the complexity of something like ‘friendship’. Life transitions and Legacy

Managing identity is an ongoing process in relation to one’s digital possessions. To this end, a great deal of HCI research has paid attention to major life events and periods of significant life transition, where people’s identity is subject to change. Odom et al. (2011, 2012, 2014) and Schoenebeck et al. (2016) focused their inquiries on teenagers, but others have considered the break-up of relationships (Herron et al., 2016; Sas and

Whittaker, 2013) or moving home (Bales and Lindley, 2013; Lindley and Wallace, 2015). However, the most significant body of work has concerned death and digital legacy, as people look to manage inherited ‘technology heirlooms’ (Banks, 2011; Kirk et al., 2010; Odom et al., 2012a) and prepare to shape their own legacy (e.g., Brubaker et al., 2014; Gulotta et al., 2014; Massimi et al., 2012, 2010; Moncur et al., 2015; Moncur and Kirk, 2014). It is beyond the scope of this literature review to consider all of these in detail. Legacy in particular is only a tangential concern for this thesis. There are, however, some common threads that are worth drawing out.

Broadly, this work describes significant challenges for people in managing their digital possessions to keep up with and reflect the changes in their lives. Digital traces are potentially far vaster, and interconnected than their physical counterparts. This makes it a struggle firstly to know the extent of what possessions you have, and hence to find, and untangle or disconnect them from services that continue to be used, particularly social media. Phone numbers can be deleted, and belongings thrown out, but it is a much more laborious, even impossible, ordeal to process all evocative emails or photographs posted on Facebook. Further as Schwarz (2014) points out, our encounters with such material can be all the more impactful when they are unexpected, presented by an unwitting algorithm, unaware of the significant change in circumstances.

The other fundamental challenge in considering ‘The Future of Looking Back’ (Banks, 2011) is recognising what will be meaningful to remember in the future, and for whom. In some respects, this uncertainty is what underlies the ‘total capture’ mentality. Some

people seek to avoid placing a burden on future generations; but for others, perhaps it is that they fail to appreciate how the unremarkable and the mundane may well become remarkable once ‘displaced’ (Radley, 1990) from present cultures and routines. Gulotta et al. (2013) report that people have mixed feelings about the potential longevity of their digital traces. There are competing desires to remove unflattering or revealing

information, meanwhile appreciating that it is just such information that they themselves lack about their ancestors, and may well become most compelling. In this sense, it is not only that digital possessions should keep up with one’s present identity, but that it might also be managed with an eye to future selves.

Narrative and Context

Concerns about legacy exemplify how people recognise the role of possessions in telling stories about ourselves and others. They anticipate “the record being interrogated for what it showed about the kinds of people that they, and those they associated with, were like” (Lindley et al., 2009b, p.6). Kirk and Sellen described many objects that were on display at home were intended to be ‘talking points’. Lindley et al. (2009b) once again emphasise that the constructing of narrative, particularly mediated by object or media is a collaborative and situated activity, oriented to present concerns:

The performances we observe as people recount their experiences to us, normally in the presence of other participants, reflect the identities they construct for

themselves at these moments. (Lindley, 2009b, p.6)

Somewhat earlier, (Frohlich et al., 2002) described ‘photo-talk’. They noted how

reminiscing with a photograph, where everyone experienced the original event, entails the emphasis of only a few particular details and some clarification, but often without a definitive resolution of a narrative. Bamberg and Georgakopoulou, (2008) emphasise the importance of such ’small stories’ in everyday conversation – in contrast to the grand autobiographical narrative or monologue. Narratives are not then fixed, or even ever complete: stories develop; they are retold with differing emphasis; the audience and storyteller orient and reorient themselves over time. People increasingly document the world around them to tell a particular story, and for this to be shared, narratively, in the moment. However, in making sense of an accumulation of digital possessions, Lindley et

al. (2009b) show how narratives are also developed post-hoc, and reflect what appears to be interesting, or important, at that present time.

It’s notable that much of the discursive work entailed in constructing a narrative involves putting things into context. Schoenebeck et al. (2016) argue that digital possessions such as Facebook posts should maintain ‘temporal integrity’ and users are required to have a degree of ‘temporal literacy’ to recognise how content is a product of its particular time. Hence, contextualisation is not only achieved discursively, but (especially online), through a range of organisational and personalising actions, which situate digital