Memory Processes

Researchers have tried to improve our memory capacities with various tech- niques. These typically target one’s short-term memory, trying to expand it with the help of constant training of memorizing numbers and/or card decks, or sim- ply with verbal rehearsal (i.e., repeat information aloud to oneself)[118, 135]. However, when it comes to enhancing episodic and semantic memory recall, the challenge lies in improving the recall of specific episodic and semantic memories,

29 2.1 Human Memory Visual stream Semantic stream Acoustic stream Attention Rehearsal loop Short-term memory Rehearsal Retrieval Long-term memory Sensory memory Recall Sensory input

Figure 2.3. Encoding, storing and retrieving memories based on the multi-store model. Sensory input arrives one’s sensory memory in the form of stimuli. Sensory memory encodes these stimuli in memory compatible forms such as visual, acoustic, semantic or tactile memories. Notice the role of attention reserves in regulating which memories reach our short-term memory. Then, subsequent rehearsals of those memories renders them as "permanently" stored in one’s long-term memory.

not overall memory capacity, and thus a different approach should be considered. First though, we need to understand how memories are being stored and later recalled. Human memory encompasses five main processes: memory encoding,

memory consolidation, memory storage, memory recall and eventually forgetting. Encoding. Information arrives to our sensory memory via our five senses (i.e., our perception), but before it can move on to our short-memory system and further, it first needs to be encoded in a form that our memory can handle (see Figure 2.3). There are three primary memory forms3in which information can be encoded: visual, acoustic and semantic. Long-term memory, where episodic and semantic memory components reside, handles memory information of all three types. After perceived information is thus encoded into visual, acoustic and/or semantic in our sensory memory, it can be stored in our short-term memory (see Figure 2.3). Memories are assumed to be encoded and subsequently stored by a sparse population of neurons in the form of "engrams" or "memory traces" [122,

30 2.1 Human Memory

148].

Consolidation. According to Atkinson and Schiffrin’s multi-store model, our attention is critical for successfully registering incoming information to our short- term memory, and the more we rehearse this information (i.e., retrieve it from our short-term memory), the more likely it becomes "permanently" stored in our long-term memory[6], evoking a process known as consolidation. During con- solidation, memories are stabilised by reducing any alterations in living neural tissue that corresponds to the engram of the memory under consolidation. [159] [92, 93, 99]. The consolidation process fires synchronously and repeatedly a group neurons (as a pattern of neural network activity, which can include the modification of synapses or creation of new ones among the neurons), making them permanently sensitized to each other, and thus more prone to activate to- gether in the future (by a process called as long-term potentiation [26]). The process consists of synaptic consolidation (within the first few hours after en- coding) and system consolidation (over a period of weeks to years). Memory consolidation processes are known to manifest during sleep and are particularly beneficial for learning[82].

Storage. After a memory has been successfully encoded and sufficiently con- solidated, it is presumed to be stored to the long-term memory in the form of engrams [122, 148]. As information flows from sensory to short-term and fi- nally to long-term memory, it gets filtered for avoiding information overload. The more a memory is recalled (e.g., through rehearsal), the more likely it is to be retained in the long-term memory because of triggering subsequent consolida- tion processes (see Figure 2.3). However, memories are innately dynamic, since the very activation of a pattern of neural activity due to recall may modify the engram itself, resulting in registering new information to the memory (modifying how the original life experience is remembered)[158].

Recall. For successfully recalling (or retrieving) a past experience or prior knowledge, our memory system utilizes engrams (i.e., memory traces) and en- coded contextual information that arrives in our short-term memory as input from our sensory memory and via our perception (see Figure 2.3). The contex- tual information may arrive explicitly (intentionally – e.g., one reviewing one’s family album for reminiscing) or implicitly (unintentionally – e.g., the smell of bee-wax candles reminding one of past Christmas celebrations). This contextual information is basis of what is known in Psychology as memory cue. Contextual information is what we can perceive with our five senses and can be anything from pictures, locations, sounds to smells, tastes, and haptic feedback. For exam- ple, in order to support patients suffering from amnesia, one can explicitly target the triggering of their episodic memory by providing such relevant contextual

31 2.1 Human Memory

information in the form of browsing photo albums. This process of replaying previously recorded relevant memory cues for triggering one’s episodic memory recall is called cued recall[35]. Cued recall can help elicit episodic and semantic memories for healthy individuals too. For example, one viewing a picture from one’s childhood could trigger the episodic recall of a long-stored memory from that time (e.g., swimming at the sea with parents during summer vacations).

Forgetting. However, it is often the case that the recall of a memory may fail. Forgetting is considered as a temporary or permanent inability to recall a piece of information previously stored in the brain, due to a mismatch between memory cues and the encoding of the information (incorrectly or incompletely encoded memories, and/or problems with the recall process). However, the memory may still be stored but rendered inaccessible [225]. This may be due to a memory disorder (e.g., Dementia) or simply due to natural loss of information over time. Memory attenuation over time has been described by a plethora of so-called "for- getting functions" that attempt to best approximate the empirical Ebbinghaus’ forgetting curve[242] (see Figure 2.1). In that respect, perhaps the most promi- nent forgetting function is the Wickelgren’s power law[244]:

m= λ(1 + β t)−ψ, (2.1)

where m is memory strength, and t is time elapsed since encoding (i.e., the retention interval). The equation has three parameters: λ is the state of long- term memory at t = 0 (i.e., right after the encoding), ψ is the rate of forget- ting, and β is a scaling parameter [248]. Forgetting is typically viewed as an unwelcome side-effect of memory rather than a memory process. Nevertheless, forgetting may be critical for our memory in filtering information overload, re- covering from negative experiences[201], forming new memories and acquiring new knowledge[55]. Schacter has rigorously described forgetting in "The Seven Sins of Memory", namely transience, absent-mindedness, blocking, misattribution,

suggestibility, bias, and persistence[202]. We briefly describe them next:

1. Transience stresses the fact that memory attenuation can vary from naturally gradual (i.e., long-term) to quite rapid (i.e., short-term). Typically, memories that are not retrieved or rehearsed may slowly dissipate over time.

2. Absent-Mindedness describes the lack of attention during encoding (i.e., "shallow encoding") or retrieval, occurring when information is superficially processed. Absent-mindedness during retrieval is expressed as forgetting to carry out a particular task or function, and thus are typically referred as fail- ures of the prospective memory.

32 2.1 Human Memory

3. Blocking refers to the phenomenon during which a deeply encoded memory is temporarily inaccessible (i.e., retrieval block), and is known to manifest both in episodic and semantic memory. Tip-of-the-tongue (TOT) state is the most prevalent example of blocking and is believed to occur due to the provision of memory cues that are related to a previously recalled memory (i.e., "part-set cuing" effect). Blocking is known for exacerbating with ageing.

4. Misattribution highlights the situations when a memory is accessible but is erroneously ascribed to an incorrect context (i.e., place, time, person, and activity). An explanation may be that one solely relies on the general semantic features of the incorrectly recalled memory.

5. Suggestibility refers to the tendency to incorporate information by others during recall (i.e., misleading questions), and is closely related to misattri- bution. In fact, it is possible to implant false episodic memories by utilizing diverse suggestive procedures (e.g., hypnosis).

6. Bias postulates that memory encoding and retrieval are highly susceptible to pre-existing knowledge and beliefs. This happens due to the natural tendency for showcasing a consistency between past and present attitudes, beliefs, and feelings (i.e., consistency/retrospective bias — see also Festinger’s theory of "Cognitive Dissonance"[76]).

7. Persistence involves the recall of an episode that one would prefer to forget (e.g., a traumatic experience). In fact, traumatic memories can often be more disrupting than forgetting per se, due to their emotional bearing. Emotion has been associated with vivid recollections due to the intervention of the "amygdala", a brain area responsible for regulating emotion and emotional memories.

In this thesis, we utilize the memory triggering potential of contextual in- formation (e.g., pictures, videos, text, etc.) for the selection and generation of memory cues. Memory cues are then replayed at random or predefined mo- ments on one’s personal devices (e.g., smartphone) for augmenting one’s mem- ory about a past experience or prior knowledge. In particular, our work draws on intentionally and repeatedly prompting memory recall processes about a tar- geted memory (i.e., memory subjected to augmentation), for implicitly evoking memory consolidation processes corresponding to the targeted memory. As we saw above, repeated instances of memory recall-consolidation processes lead to permanently storing a memory. This phenomenon comprises the essence of cued

recall, the theoretical underpinning of this work. Due to the potential of (mobile) cue-based memory augmentation in forming strong memories, we believe that the memory augmentation effect will persist through time, even without further