This article was downloaded by: [ ] On: 20 May 2012, At: 02:49
Publisher: Routledge
Informa Ltd Registered in England and Wales Registered Number: 1072954 Registered office: Mortimer House, 37-41 Mortimer Street, London W1T 3JH, UK
Philosophical Psychology
Publication details, including instructions for authors and subscription information:
http://www.tandfonline.com/loi/cphp20
Rich experience and sensory memory
Elizabeth IrvineAvailable online: 04 Apr 2011
To cite this article: Elizabeth Irvine (2011): Rich experience and sensory memory, Philosophical Psychology, 24:2, 159-176
To link to this article: http://dx.doi.org/10.1080/09515089.2010.543415
PLEASE SCROLL DOWN FOR ARTICLE
Full terms and conditions of use: http://www.tandfonline.com/page/terms-and-conditions
This article may be used for research, teaching, and private study purposes. Any substantial or systematic reproduction, redistribution, reselling, loan, sub-licensing, systematic supply, or distribution in any form to anyone is expressly forbidden.
The publisher does not give any warranty express or implied or make any representation that the contents will be complete or accurate or up to date. The accuracy of any
Vol. 24, No. 2, April 2011, 159–176
Rich experience and sensory memory
Elizabeth Irvine
One of the possible ways to explain the experience of visual richness is to posit a level of nonconceptual or phenomenal experience. The contents of this level of experience have recently been equated with the contents of sensory memory. It will be argued that sensory memory cannot provide these contents along two broad points. First, the conception of sensory memory relied on by these authors conflates the phenomena of visible and informational persistence, and makes use of an outdated ‘‘iconic’’ model of visual short-term memory. Second, the way in which subjects’ reports are used to show that specific unreported contents are nevertheless experienced on a phenomenal level is questioned, using evidence on gist and high-level categorical perceptual processing. It is concluded that sensory memory, properly understood, cannot provide the kind of visual content required to support a level of richly detailed phenomenal experience, or a pictorial account of perception. Finally, alternative ways of explaining visual richness are suggested.
Keywords: Phenomenal Consciousness; Sensory Memory; Subjective Report; Visual Richness
1. Introduction
Our visual experiences seem full of richly detailed objects. However, the capacity constraints on working memory mean that only a small amount of attended information can be accessed and reported at any one time. As access via some form of report is currently seen as the main or only way of probing whether particular contents are conscious (see, e.g., Dehaene & Changeux, 2004; Kouider & Dupoux, 2007), this suggests that much of the detail we think we see is actually not consciously present. In this case there is a basic problem in how to explain the experience of visual richness.
What will be called the ‘‘rich’’ view suggests that pre- or unattended contents, while not accessed or reported, are still experienced in a rich and detailed way. It is argued that unreported visual information is experienced on a different ‘‘level’’ to
Correspondence to: Elizabeth Irvine, Department of Philosophy, University of Edinburgh, Dugald Stewart Building, 3 Charles Street, Edinburgh, EH8 9AD, UK. Email: [email protected]
Elizabeth Irvine is a PhD student at the University of Edinburgh.
ISSN 0951-5089 (print)/ISSN 1465-394X (online)/11/020159-18ß2011 Taylor & Francis DOI: 10.1080/09515089.2010.543415
reported visual information. Information about attended and reported objects/areas is ‘‘cognitively accessed’’ (Block, 2001, 2005, 2007), and is experienced on a ‘‘fact’’ (Dretske, 2004, 2007) level of awareness. In contrast, information about unattended and unreported objects/areas is not cognitively accessed but is experienced on a lower ‘‘phenomenal’’ (Block) or ‘‘object’’ (Dretske) level of awareness. Similarly, Tye (2006, 2009) argues that the unreported visual detail present in attended regions of space is represented nonconceptually. A recent move to identify the locus of this lower level of nonconceptual or phenomenal experience makes use of the results of partial report paradigms (e.g., Sperling, 1960). These paradigms show that more information than subjects can concurrently report is stored in a short-term visual (sensory) memory. As this is consistent with the idea of phenomenology ‘‘overflowing’’ cognitive access, many authors have recently claimed that the rich contents of sensory memoryarethe contents of unreported and unaccessed phenomenal experience (see Block, 2007; Fodor, 2007, 2008; Jacob & de Vignemont, 2010; Tye, 2006). This claim will be examined by focusing on two main questions.
The first question is to what extent the results of partial report tasks really support the claims of the rich view. It will be argued that not only are partial report tasks largely irrelevant to discussions about the contents of visual experience, but they can also be used to show that sensory memory does not provide the sort of coherent, detailed and precategorical representations of which phenomenal experience is supposed to consist. Instead, sensory memory is made up of stores of different kinds of information, some spatial and some categorical, each subject to different rates of information processing and decay. This ensures that the main question of the debate, whetherthesensory memory (see, e.g., Tye, 2006, pp. 511–513) ortherepresentation (see, e.g., Block, 2007, section 9) of an unreported object is phenomenally present, is confused and misleading, as it is not clear how to identify ‘‘the’’ representation of an object given the distributed nature (over both brain areas and time) of visual processing.
The second question concerns the relationship between partial report tasks and the contents of subjects’ experiences during the tasks. The phenomenon of partial report superiority, in itself, concerns the properties of short-term visual memory only, so can tell us nothing about what subjects experience. Instead, the crucial link between partial report tasks and claims about phenomenology is through subjects’ reports of what they experience. The second question therefore concerns to what extent reports of visual richness depend on the processing of detailed visual information. It will be argued that such reports are in fact often based on scene-level or gist information, not on detailed item-specific information. Therefore even if there are large stores of detailed sensory information in the visual system, subjects’ reports of visual richness are not sufficient to show that this detailed information is experienced on a phenomenal level.
The answers to both of these questions seriously undermine the currently popular claim that the contents of sensory memoryarethe contents of a richly detailed level of phenomenal experience. Instead they provide further evidence that the current neuropsychological understanding of the visual system is simply not consistent with a
pictorial account of perceptual consciousness, some contents of which are accessed and some not. Some suggestions will be made on how to better explain the appearance of visual richness as a product of different perceptual, cognitive and motor capacities used and enabled in different stages of visual processing as they occur over time.
2. Partial Report Superiority
In order to claim that detailed but unreported representations of scenes are experienced on a phenomenal or nonconceptual level, supporters of the rich view often refer to the partial report paradigm (Landman, Spekreijse, & Lamme, 2003; Sligte, Scholte, & Lamme, 2008; Sperling, 1960). Aside from appeals to intuition, this paradigm provides the primary evidence in favor of the rich view and its corresponding taxonomy of phenomenal–nonconceptual and accessed–conceptual visual consciousness. The paradigm suggests that a short-term, large capacity sensory memory stores detailed visual information. Only some of this information can be accessed via attentional selection, and most of this information is quickly ‘‘forgotten.’’ However, it is argued that this unaccessed content could provide subjects with a lower phenomenal level of visual experience. The original paradigm and modern variations are described below.
Sperling (1960) claimed to establish the existence of a precategorical ‘‘iconic’’ memory store through the phenomenon of partial report superiority. In his classic paradigm, subjects are shown a display of letters for a short time (15–500 ms), followed by a variable delay, and asked to report as many letters as they can (see Figure 1). Subjects in this ‘‘full report’’ condition are capable of reporting around 4.5 letters out of 12 letters, which is reasonably invariant to the duration of the display. In the partial report condition, a display of letters are again shown for a short time, and now a cue for a particular row is given some time before or after the stimulus (e.g., from 50 ms before onset up to 1000 ms after stimulus offset). The cue can either be visual, such as an arrow, or an audio cue, with high, medium and low tones cuing top, middle and bottom rows, respectively. After the cue there is a further variable delay before the subject responds. In this condition, subjects report as many letters as they can from the cued row, and on average report 3.03 letters from each 4-item line. Sperling suggested that since subjects reported 3.03 out of 4 lettersfor any cued row, information about 9.01 items from the 12 letters must be available at the time of the cue, far higher than is suggested in the full report condition. Subjects also report seeing all the letters in the display, despite not being able to report all of their identities. Sperling suggested that a form of short-term ‘‘iconic’’ memory explains these results. Iconic memory could store large amounts of low-level precategorical information for brief periods of time, but only a small amount of it would be fully processed and reported at any one time. This explains how information from any cued row could be accessed, but the short duration of this form of memory ensures that information from noncued rows would decay before subjects could report it.
More recent versions of this paradigm suggest that short-term visual memory has a much larger capacity and longer duration than Sperling found. Landman et al. (2003)
and Sligte et al. (2008) used experimental paradigms that combine aspects of the partial report and change blindness paradigms. Landman et al. showed subjects 8 differently oriented (and differently oriented and sized) rectangles for 500 ms, followed by a delay, followed by a display in which one of the rectangles had changed. In the full report condition subjects showed poor performance, consistent with change blindness. However, when cues that mark particular rectangles are shown between the initial and changed displays, performance increases dramatically. Subjects’ performance in the cued condition suggests that with practice they can maintain information about 6–7 out of 8 rectangles over 1.5 s after stimulus offset. Sligte et al. (2008) took this paradigm further and used 32 oriented rectangles in a similar change detection task. Although the strength of afterimages, possible chunking effects (see Brockmole & Wang, 2003), and the use of recognition rather than recall in these tasks may play a significant role in determining their results, they also found evidence for a larger capacity and longer lived form of sensory memory than found in the original paradigm (for detailed discussion and alternative explanation of these results, see Phillips, in press).
The results of these partial report tasks are consistent with, and seem to provide empirical support for, the rich view. There is a kind of visual memory that contains large amounts of detailed information that is not all concurrently reportable, and is available for only short periods of time. Subjects’ reports indicate that this information is experienced. The contents of this form of memory could therefore provide the contents of a rich phenomenal level of experience. Accordingly, supporters of the rich view have claimed that the contents of sensory memoryarethe contents of phenomenal experience.
3. The Structure of Sensory Memory
The first problem with this claim concerns the relevance of partial report superiority to claims about phenomenology, and how the structure of sensory memory is
Figure 1 Partial report condition. Full report condition does not include the cue
(Sperling, 1960).
inconsistent with the supposed contents and character of phenomenal experience. By comparing the ‘‘iconic’’ model of sensory memory that proponents of the rich view rely on with the more fragmented current model, it will be shown that partial report superiority fails to provide evidence for the existence of a rich level of phenomenal experience.
3.1. The Rich View and Iconic Memory
The original model of sensory memory used to explain the phenomenon of partial report superiority was a form of ‘‘iconic’’ memory. It was characterized as a short-lived, precategorical memory from which limited amounts of information could be accessed at any one time. It suggested that a persisting picture or ‘‘icon’’-like representation existed in memory, from which further processing could provide more categorical and conceptual information. Although this model is now more than thirty years out of date, it will be shown that many authors continue to rely on the language and properties of iconic memory to argue for a full phenomenal experience of a display. The problems in relying on this kind of model of sensory memory are detailed in the following section.
Some proponents of the rich view explicitly refer to iconic representations in their arguments about the contents of experience. For example, Fodor (2008) states that: ‘‘The argument just set out [from partial report superiority] is empirical; it suggests that there is iconic representation in perception. . .’’ (p. 189, followed by more evidence for this view). Dretske (2006) echoes this: ‘‘. . .subjects extract [letter identity] information from what they describe as a conscious but rapidly fading image (‘‘icon’’) that persists for a short time after removal of the stimulus’’ (p. 175). Although some authors are aware of more current models of sensory memory, they too attribute the properties of iconic memory to the contents of what they apparently view as non-iconic sensory memory. For example, in his discussion of partial report paradigms, Tye (2006) assumes that the contents of sensory memory provide sufficiently detailed representations for item categorization to occur at a later stage of processing: ‘‘So long as information is present about the contents of the unattended rows on the basis of which shape and letter identifications could have been made. . .the relevant visual experiences are representationally rich’’ (p. 511). In his discussion of Landman et al. (2003), Block (2007) is more explicit in how he conceives of the item-specific representations in sensory memory, and states that: ‘‘The subject has persisting experiences as of more specific shapes than can be brought under the concepts required to report or compare those specific shapes with others’’ (p. 489).
The representations in sensory memory that these authors refer to are assumed to be detailed and contain only low-level information (letters may be categorized as letters, but are not identified as specific letters). They state that in order to identify the letters (or the orientation of rectangles) in a display, further processing of this low-level information is necessary. Further, it is assumed that sensory memory contains coherent and unified representations that can easily be referred to as the
memory of an array, orthespecific representations of items. The representations in sensory memory that the rich view makes use of are therefore largely of the same format as iconic representations. They are (largely) precategorical or preconceptual, and provide a coherent and detailed (pictorial) base for further conceptual processing.
3.2. Visible and Informational Persistence in Sensory Memory
The characterization of the contents of sensory memory made by proponents of the rich view is based on an outdated understanding of the structure of sensory memory. This gives rise to serious problems when they equate the contents of sensory memory, uncovered in partial report paradigms, with a richly detailed level of phenomenal experience. Instead of providing a short lived, visually detailed but nonconceptua-lized ‘‘iconic’’ base for further processing, sensory memory is instead an umbrella term referring to different kinds of persistence and different kinds of memory stores, (some containing identity information), all of which develop and decay at different rates. In contrast to the original ‘‘one-store’’ model of iconic memory, current models of sensory memory refer to the different phenomena of visible and informational persistence (see Coltheart, 1980, for original distinction; for discussion and reviews, see Loftus & Irwin, 1998; Luck & Hollingworth, 2008), and the division of informational persistence into visual and nonvisual stores (for original distinction, see Irwin & Yeomans, 1986). Although visible and information persistence are often confusingly identified as bringing about the same perceptual effects, they are quite different. The differences between these two kinds of persistence are described below, along with the problems this raises in using partial report superiority as evidence of a rich phenomenal level of experience.
Visible persistence refers to continuing activity (typically less than 100 ms duration) found in early visual areas after stimulus onset, and generates visual afterimages (e.g., lightning in a dark sky). Visible persistence is relevant to discussions of phenomenology as it refers to both the content and the duration of particular experiences. It is often assessed using temporal integration tasks (see Coltheart, 1980; Di Lollo, 1980; see also Luck & Hollingworth, 2008, pp. 17–19, for a summary of alternative paradigms to assess visible persistence). In Di Lollo’s (1980) temporal integration task, two arrays of dots in a 5 by 5 matrix are presented sequentially in time, superimposed in the same spatial position. One dot is ‘‘missing’’ from the superimposed matrix and its position has to be identified. For short presentation durations and inter-stimulus intervals, persisting activity in early visual areas enables individual arrays to be processed and experienced as a single array, allowing for an easy identification of the ‘‘missing’’ dot.
However, the duration of visible persistence is so short that it is often over even before a cue is presented in partial report tasks, ensuring that identity information is not ‘‘read off’’ from a persisting sensory image of the type assessed in temporal integration paradigms. Instead, partial report paradigms assess the persistence of information(including letter identity), and cannot be used to make claims about the
content or duration of experiences. Information persists in two different ways: in a visible analog representation, and in a nonvisual identity code (Irwin & Yeomans, 1986). Irwin and Yeomans found that a visible analog representation preserves shape and location information and is stored for 150–300 ms after stimulus offset. A nonvisual post-categorical store preserves abstract information such as identity for up to 500 ms (Di Lollo, 1980; Mewhort, Campbell, Marchetti, & Campbell, 1981).
Partial report tasks are used to assess how long both the visible analog and post-categorical stores are preserved. For short delays between the display and the cue (up to 300 ms) subjects are able to identify most of the letters in any cued row, as both kinds of information are available. For longer delays between the display and the cue (between 300–500 ms) spatial information is lost as the visible analog representation decays, so subjects routinely make ‘‘location errors.’’ Although they can identify some letters from the display, they cannot assign them to a particular row.
The basic distinction between informational and visible persistence generates a significant problem for the argument that partial report superiority provides evidence of a richly detailed level of phenomenal or nonconceptual experience. As Luck and Hollingworth (2008) state, partial report tasks cannot be used to make claims about phenomenology: ‘‘the partial-report technique does not measure directly thevisible aspect of visual sensory memory, but rather thatinformation persists after stimulus onset’’ (p. 16, original italics). Partial report superiority illustrates the persistence of information in memory. It does not illustrate the persistence of information in experience, as this refers to the different phenomena of visible persistence which cannot be assessed by partial report tasks. Empirical data from partial report tasks, by itself, does not contain any information about the content or duration of the phenomenal states subjects experience when they view displays, only the content and duration of information stored subsequent to viewing the display. Partial report superiority therefore provides no evidence that subjects experience any of the detailed unreported information that persists in memory.
This distinction between visible and informational persistence is not noted by many of the supporters of the rich view, for whom it is a significant problem. However, some authors have tried to argue that the persisting information uncovered in partial report tasks is also experienced on a phenomenal level. It will be shown that their arguments fail as they are based on a feature of older models, mentioned above, that sensory memory refers to a single memory store.
3.3. One-Store, Two-Stores
Both Block (2007) and Tye (2006) construct arguments that are supposed to show that the persisting information that enables partial report superiority is also experienced through visible persistence. These arguments are based on the assumption that visible and informational persistence are different aspects of the same memory store.
Block clearly assumes that visible and informational persistence are properties of the same basic memory state, and in reference to Landman et al. (2003) states: ‘‘Subjects are apparently able to hold the visual experience for up to 1.5 seconds—at least ‘partial report superiority’ (as it is called) lasts this long’’ (Block, 2007, p. 488). As shown above, partial report superiority is not a measure of the duration of visual experience, only the persistence of information. Block then goes on to detail experiments concerning visible persistence as evidence that persisting information is experienced on a phenomenal level, relying on the assumption that visible persistence is also a property of persisting information (see, e.g., pp. 490–491, 494, 532). In doing so, Block even cites original papers that argue against the conflation of informational and visible persistence as aspects of the same single memory store (e.g., Coltheart, 1980; Di Lollo, 1980).
Likewise, Tye (2006) acknowledges that partial report paradigms examine the availability of information only, but argues that experiments concerning visible persistence show that persisting information is experienced. He does this by assuming that there is one sensory memory that can be assessed by both partial report tasks, uncovering the informational part, and by temporal integration tasks, showing that the sensory memory is experienced after the stimulus has disappeared (see Tye, 2006, pp. 511–513). He subsequently concludes that ‘‘the visual memory system operates at the level of visual appearances’’ (p. 511), ensuring that the information that enables partial report superiority is phenomenally conscious.
The descriptions of visible and information persistence given above should make it obvious why these arguments cannot work. Visible and informational persistence are properties of different stages of visual processing. Visible persistence refers to the decaying trace of stimulus-related activity in early visual areas, and disappears well before persisting information does. Informational persistence refers to the ongoing processing and short-term storage of information in later cortical areas, including identity information. Experimental paradigms concerning visible persistence are simply irrelevant to the discussion of whether persisting information is experienced, as they measure a property of a much earlier stage of visual processing. Purely from a consideration of the different times for which visible and informational persistence occur, it is clear that they are not two aspects of the same single store (see also Coltheart, 1980).
Given a better understanding of what partial report paradigms are designed to assess, it can be seen that partial report superiority, as an example of informational persistence, is not relevant to questions about the content and duration of visual experience. Further, experimental work on visible persistence cannot be used to show that persisting information is also experienced. Therefore partial report superiority cannot be used to argue that the detailed contents of sensory memory are experienced on a phenomenal level.
Having exposed these confusions about the structure of sensory memory, it remains to be seen if the contents of sensory memory can give us any guide about the contents of experience. The following section relates the problem in establishing what ‘‘the contents of sensory memory’’ refers to. If there is no clear referent of
‘‘the sensory memory’’ or ‘‘the specific representation’’ of an object, then the question the rich view tries to answer, whether these memories or representations are phenomenally conscious, is unanswerable. It will be argued that the dynamic contents of sensory memory do not provide such representations, undermining the motivating question for this debate about the contents of experience.
3.4. What Representations?
Another serious problem for the claim that the contents of sensory memory are the contents of phenomenal experience is that sensory memory does not generate or preserve easily identifiable representations of a display or its contents. This section builds on the fragmented nature of sensory memory, with different aspects developing and decaying at different rates. The contrast between the dynamically constituted contents of sensory memory and our apparently coherent and unified experiences suggests that the contents of sensory memory cannot simply be equated the contents of phenomenal experience. It also shows that the question of whether persisting representations of unreported items are experienced is a badly posed question, based on mistaken assumptions about the nature of visual processing.
The way that different kinds of persisting information develop and decay in sensory memory illustrates difficulty in pinpointingtherepresentation or memory of the contents of a display. For example, scene-level information (gist) is available very rapidly, as it can be processed from stimulus durations as short as 20 ms, and can be completed in less than 100 ms from stimulus onset (see, e.g., Oliva, 2005; Van Rullen & Thorpe, 2001). In contrast, it takes much longer to populate the post-categorical store with identity information. Item specific processing starts around 100 ms with letter-related processing occurring around 150 ms, and high-level case independent representations present after 220 ms (Tarkiainen, Cornelissen, & Salmelin, 2002; Thorpe, Fize, & Marlot, 1996). This information decays between 300–500 ms after stimulus offset, in contrast with the different decay rate of the visible analog store, which ends 150–300 ms after stimulus offset. As described above, this difference in decay rates results in subjects making location errors for longer display-cue intervals. From gist processing to spatial processing and letter identification, through different decay trajectories, the contents of sensory memory radically change over short time scales. However, subjects do not report experiencing this change. While the contents of sensory memory are the sequential products of different levels of visual processing, the contents of visual experience are not. This is reflected in subjects’ reports that they read letter identities off ‘‘a’’ memory of the array (see, e.g., Shanahan & Baars, 2007, p. 525). Given that sensory memory does not provide the kind of stable representations that map onto what subjects claim to experience, there is clearly a significant problem in claiming that contents of sensory memory are the contents of phenomenal experience.
An attempt to more closely map the stages of visual processing with visual phenomenology has been criticized on just these phenomenological grounds. Many commentators on the interpretation of partial report superiority offered by the rich
view suggest that the letters of a display are experienced only via category information (giving an illusion of visual richness) before they are attended to and then experienced as specific letters (see replies to Block, 2007, by Burge, Grush, Jacob, Kouider et al., Levine, Naccache & Dehaene, Papineau, Spener and Van Gulick). However, Block argues that this entails a shift in phenomenology which seems inconsistent with experience:
If there is only generic phenomenology before the cue, and if the cue causes the generic phenomenology to be replaced by specific phenomenology, then there is a shift from generic to specific phenomenology. The fact that subjects report no such phenomenological shift might not be strong evidence against this view, but it is some evidence. The vast literature on this topic (including two Ph.D. theses I have read) contains no mention of such a thing as far as I know. I myself can testify that even looking for such a shift, one does not experience it. (Block, 2007, p. 532)
Although it may be possible that subjects experience gist and the various stages of processing before item identity, and then the differential decay rates of spatial and identity information, this is not the standard way we would describe our visual experiences.
Together the points raised above illustrate the point that it is very difficult to isolatethespecific representations or memories of items present in sensory memory. The contents of sensory memory are distributed over different areas of visual processing, and contain different kinds of information that are constantly developing and decaying at different rates. These dynamically constituted contents of sensory memory cannot easily be equated with the coherent experience subjects claim to have. The basic question of the debate, ‘‘whether those specific representations are phenomenal’’ (Block, 2007, p. 531, original italics), is also shown to be a badly posed question. The experience of visual richness cannot simply be mapped onto easily individuated, detailed, stable, precategorical representations in sensory memory as they do not exist.
3.5. Summary
It has been argued that partial report paradigms, in themselves, provide no evidence about how subjects experience visual displays, since partial report superiority is an example of informational persistence only. It does not measure the kind of persistence that gives rise to phenomena more clearly related to visual phenome-nology, such as visual after-effects and temporal integration of sequential arrays. As an example of the way spatial and categorical information persists inmemory, partial report superiority cannot be used to support the claim endorsed by the rich view that there is a rich and largely unaccessed level of phenomenal awareness.
Partial report superiority can also be used to show that representations with iconic properties are not generated or preserved in sensory memory. Information is stored in both a visible analog representation (containing spatial/location information) and a nonvisual post-categorical store (containing object-specific category information), the contents of which develop and decay at different rates. The way that the contents
of sensory memory change constantly over short time periods also ensures that it is difficult to identify the representation or memory of the contents of a display. The question of whether ‘‘the specific representations’’ stored in sensory memory are experienced on a phenomenal level is therefore meaningless.
Data from partial report paradigms cannot be used to support the idea that sensory memory provides the contents of a constantly present rich phenomenal level of experience. The following section includes a brief discussion of a further methodological problem in generalizing the results of partial report superiority to perception in general. Even without the problems described above, the claim that there isalwaysa richly detailed level of phenomenal experience based largely on the results of standard partial report paradigms is difficult to sustain.
4. Problems with Generalization
Further to claims that subjects experience all the letters or rectangles in a simple display, partial report superiority has been used to argue for a more pervasive richness of phenomenal experience. For example, partial report superiority is often used to argue for the rich experience of scenes in change blindness paradigms, despite subjects often being unable to report changes between scenes (e.g., Block, 2007, pp. 487–488; Dretske, 2004, 2007; Tye, 2006, pp. 514–515). However, the letter and rectangle displays used in partial report tasks are simple and sparsely distributed. Natural scene perception operates under very different conditions. When viewing displays containing many closely packed stimuli, perceptual crowding and lateral masking can prevent easy identification or discrimination of stimuli (e.g., see Pelli, Palomares, & Majaj, 2004; Wertheim, Hooge, Krikke, & Johnson, 2006). These two phenomena are thought to be the product of constraints on early pre-attentional visual processing, and show that there are some basic limits on the amount of rich and detailed information that can be extracted from natural scenes.
A more general constraint on the breadth of the richness of visual experience is how much of a scene can be fixated and attended to. Although attention is sufficient to encode some aspects of visual detail from natural scenes into even long-term visual memory (Hollingworth & Henderson, 2002), visual detail from nonfixated areas or items is unlikely to be processed due to lower resolution of information processing. Some authors comment explicitly on the limits of visual processing for unattended items, but continue to claim that the results of partial report superiority for simple attended arrays also generalize to the total area of much larger and more complex arrays, such as those found in change blindness (e.g., Dretske, 2007, pp. 217–218; Tye, 2006, p. 514).
In fact, there is some evidence that partial report superiority is not found in natural scene perception. In contrast with the results of partial report tasks used by Landman et al. (2003) and Sligte et al. (2008), Liu and Jiang (2005) found that using a comparable recognition task for items in a natural scene (250 ms stimulus duration, 1000 ms display-cue interval), performance was very low (0.7 items recognized
compared to apparent capacity of up to 32 items in Sligte et al., 2008). The authors comment that ‘‘although 250 ms may be sufficient for people to perceive semantic gist of a scene. . .it is not long enough for most visual details to enter VWM [i.e., to be reported post-cue]’’ (Liu & Jiang, 2005, p. 656).
While this paper does not seek to support a particular theory of the character of unattended visual information, there are reasons to suppose that the phenomenon of partial report superiority for sparse and simple arrays cannot be used to argue for the existence of a richly detailed level of experience for natural scenes. The constraints on visual processing of larger and more complex scenes suggest that unreported and unattended visual information is not always available in the same detailed format as for simpler arrays.
The following section continues the investigation into how visual richness is or is not constituted, and questions the only real link between partial report paradigms and phenomenology: subjects’ reports of visual richness. These reports are clearly used as evidence in partial report tasks that subjects do experience all the unreported letters, and reports of visual richness are also implicitly used to support the generalization from partial report superiority to the nature of perceptual experience in general. The use of reports of visual richness as evidence for a detailed phenomenal level of experienced is assessed below.
5. Reports of Visual Richness and Gist
It was argued above that partial report paradigms provide evidence only about information persisting in short-term memory, not about what subjects experience. Instead it is subjects’ reports of seeingallthe letters orallthe rectangles that provide the essential link in arguing that the unreported contents of sensory memory are experienced on a rich phenomenal level. This link is investigated below to assess whether the processing of visual detail is necessary to generate reports of visual richness. Instead, it will be shown that reports of richness are often generated using high-level categorical or scene-level information. In this case, not even an appeal to subjects’ reports of richness provides support for the basic notion of a richly detailed level of phenomenal experience.
There are various versions of the conceptual argument that experiences and reports of richness do not depend on the presence of rich information in the brain,1 now supported by a growing body of empirical evidence. In recent years the large role played by the processing and storing of gist and high-level categorical information has been recognized both in vision and memory research, and suggests that the processing of richly detailed information isnotnecessary for the generation of reports of visual richness. In this case, subjects may not need to process or experience any item-specific detailed information in order to report that they see all the letters or rectangles in a display.
Beginning with research into memory, the Deese-Roediger-McDermott (DRM) paradigm (see Roediger & McDermott, 1995) shows how strong encoding of the
theme of a word list leads to high false recognition rates for words consistent with the list but not actually on it. Research into scene gist in visual perception also shows how the specific items that subjects report from a scene are strongly influenced by the meaning of the scene. For example, Brewer and Treyans (1981) found that subjects tended to report seeing books when they had spent some time in an office, even though none were present. This response bias towards (sometimes falsely) recognizing objects consistent with the gist of a scene is a widely found and reliable phenomenon (see, e.g., Hollingworth & Henderson 1998, 1999; Pezdek, Whetstone, Reynolds, Askari, & Dougherty, 1989), and is utilized in Castelhano and Henderson’s (2008) contextual bias paradigm. Subjects are shown a photograph of a scene for 20–250 ms followed by a 50 ms mask, and then asked whether a target object is present in the photograph. For a scene of a park, a target object might be a bench (consistent) or a refrigerator (inconsistent). Subjects tend to respond that consistent items are present and that inconsistent items are not, despite the fact that none of the target objects are actually present in the scenes. Castelhano and Henderson note:
. . .participants in this experiment were not told until debriefing that the objects
were not actually present in any of the scenes. When asked during the debriefing period whether they had suspected this during the experiment, participants typically reported that they had not noticed. (p. 665)
This extreme example of the way in which subjects rely on scene gist shows how reports about specific objects and visual detail often fail to reflect any processing of detailed visual information. The contextual bias paradigm shows that such reports can be generated even in the absence ofanyexperience of the reported specific visual detail in the scene.
Of particular relevance to the reports of visual richness generated in partial report paradigms, de Gardelle, Sackur, and Kouider (2009) have recently tested the role of scene-level information and expectation in response generation for simple letter arrays. When shown a display of letters with pseudo-letters in noncued rows, subjects still report seeing ‘‘all the letters,’’ and are surprised when told that displays contain these pseudo-letters. Scene-level category information (that the items are letter-like) combined with expectations about the contents of the display drive responses about the presence of specific objects, sometimes misleading subjects who give an incorrect description of the contents of a scene (see also Kouider & Dupoux, 2004).
Partial report paradigms are used to show the existence of a memory store of richly detailed information, but these reports of visual richness are clearly not based on the detailed item-specific contents of this store. Instead, scene-level information plays the primary role in determining the kinds of reports that are later used as evidence that detailed persisting information is experienced on a phenomenal level. The evidence above shows that reports of visual richness cannot be used in this way, and fail to provide support for the existence of a richly detailed level of unaccessed, phenomenal experience.
6. Conclusions
It has been argued that neither the phenomenon of partial report superiority nor subjective reports of visual richness provide evidence for the claim that there is a richly detailed, precategorical phenomenal level of experience. Partial report superiority provides evidence only about the persistence of information in sensory memory and in itself cannot be used to make claims about the content or duration of visual experiences. The structure of sensory memory, containing stores for both visual (spatial) and nonvisual (identity) information, developing and decaying at different rates, also fails to provide anything that can be easily identified as the representation of an item in a display. The basic question of this debate, whether specific representations are experienced on a phenomenal level, is therefore an inappropriate question to ask. Given a detailed description of the structure and properties of sensory memory the claim that the contents of sensory memory are the contents of a detailed phenomenal level of experience is difficult to defend.
Further to this, the generalization from the phenomenon of partial report superiority for simple and sparse arrays to a model of perception for much larger and more complex scenes was questioned. From this it also became clear that the only real link between partial report tasks and phenomenology were subjects’ reports of experiencing ‘‘all the letters’’ in a display, supported by more general reports of visual richness. It was therefore necessary to investigate the way in which reports of visual richness are generated. A discussion of the role of scene-level processing in generating item-specific reports showed that reports of visual richness are oftennotbased on the processing (or presence) of richly detailed visual information. Instead, gist, category information and associated expectations are used by subjects to make claims about whether or not specific items are present in a scene or display, sometimes leading to cases of ‘‘false perception.’’ The use of scene-level information to generate reports of visual richness shows that such reports cannot be used as evidence for a rich level of phenomenal conscious.
These arguments show how partial report superiority cannot be used to make claims about the contents of visual experience, and that phenomenal reports of richness can be generated without detailed visual information. Therefore, contrary to popular assumption, partial report superiority provides no support for the rich view and its claims about the existence of a precategorical but visually detailed level of phenomenal experience. The evidence presented here suggests further that the general strategy of trying to establish simple mappings between visual appearances, such as visual richness, and visual representations, is problematic. This strategy ignores the fact that different kinds of information, some spatial/location, some categorical or conceptual, some scene-level and some item-specific, develop and decay at different times. Identifying the representation that gives rise to the appearance of visual richness is impossible as it relies on an oversimplistic model of perceptual processing.
An alternative strategy is to model the richness of visual experiences as a product of the different perceptual, cognitive, and motor capacities that are used and enabled
by different kinds of processing occurring over time. These include the different abilities of detecting, categorizing, identifying, remembering and attending to objects, and the role of gist and conceptual processing in providing a spatial template for the direction of attention, and activating expectations about which items are likely to be present. The ability to saccade is also crucial to regularly and rapidly update visual information. Dehaene, Changeux, Naccache, Sackur, and Sergent (2006) suggest that the appearance of visual richness may depend on perceptions occurring actively over time, such that reports of visual richness ‘‘might arise because viewers know that they can, at will, orient attention to any location and obtain conscious information from it’’ (p. 210). Sensorimotor accounts of perception also emphasize the role that our abilities to interact with the world play in determining the way that we experience it, and suggest that visual detail does not need to be processed in order for us to experience it as a property of the world (e.g., see Noe¨, 2004; O’Regan & Noe¨, 2001). Corresponding to our different capacities to interact with a scene there will also be many different ways of being conscious of a particular object. In this case partial report superiority does not illustrate a simple distinction between accessed and phenomenal consciousness, conceptual and nonconceptual awareness, generic and specific phenomenology, or even partial and full awareness (Kouider, 2004; Kouider et al.’s reply to Block, 2007) of the letters in the display. Instead it illustrates the way that access to particular kinds of visual information changes over time, scene-level information typically being followed by the ability to recall some item-specific information from sensory memory. Subjects are clearly conscious of the letters in the sense that these letters make up the contents of the display, and subjects are aware of the category of the contents in the display. However, subjects are not able to report all the letter identities (all at once), though they can report a few in a small time window. The question of whether subjects are conscious of the unreported letters is currently unanswerable as it is not clear what kind of informational access or capacity should be deemed sufficient to count as consciousness of a particular item. There may be no nonarbitrary answer to this, but this question and others like it will remain meaningless until further specification of a particular conscious capacity (e.g., item identification) is made.
Given its dependence on current cognitive science, this alternative approach to understanding visual consciousness makes a few radical suggestions about what kinds of research questions to pursue. It suggests that visual richness should not be characterized as a feature of a single visual representation, but as a product of many integrated capacities to react towards a scene, enabled progressively over time. In this case, the continuing investigation of the different kinds of capacities enabled under different viewing conditions (e.g., stimulus duration, array size, masking), and the relationships between these capacities and the contents of subjects’ reports (e.g., reports of richness, confidence ratings), will further detail the way in which visual information appears to us as subjects. For example, as discussed above, research into gist perception has already uncovered the power of scene-level information in determining responses about the presence or absence of specific objects. Research into the role of saccades in the perception of natural scenes, properties of spatial and
object based attention, connections between sensory and motor areas, and the way in which expectations shape visual processing and experience will all provide a more detailed account of the way in which active visual systems determine the nature of visual experiences. Instead of seeking to find simple mappings between visual appearances and visual representations, modeling a wide range of dynamic perceptual capacities should provide more insight into how our rich visual phenomenology is constituted.
Acknowledgments
Many thanks to Andy Clark, John Henderson, Ned Block, and the Edinburgh PPIG group for discussion and comments on earlier versions of this paper. This work was carried out while being supported by a PhD Scholarship from the British Society for Philosophy of Science.
Note
[1] Dennett (1991) illustrates this misconception in his example of viewing Marilyn Monroe wallpaper (p. 355), Block (2007) calls it the ‘‘photographic fallacy’’ (p. 533), and Van Gulick (2007) refers to it as the ‘‘movie screen of the mind’’ model of consciousness (p. 529).
References
Block, N. (2001). Paradox and cross purposes in recent work on consciousness. Cognition, 79, 197–219.
Block, N. (2005). Two neural correlates of consciousness.Trends in Cognitive Science,9, 46–52. Block, N. (2007). Consciousness, accessibility, and the mesh between psychology and neuroscience.
Behavioral and Brain Sciences,30, 481–548.
Brewer, W. F., & Trevans, J. C. (1981). The role of schemata in memory for places. Cognitive Psychology,13, 207–230.
Brockmole, J. R., & Wang, R. F. (2003). Integrating visual images and visual percepts across space and time.Visual Cognition,10, 853–873.
Castelhano, M. S., & Henderson, J. M. (2008). The influence of color on the activation of scene gist. Journal of Experimental Psychology: Human Perception and Performance,34, 660–675. Coltheart, M. (1980). Iconic memory and visible persistence. Perception and Psychophysics, 27,
183–228.
de Gardelle, V., Sackur, J., & Kouider, S. (2009). Perceptual illusions in brief visual presentations. Consciousness and Cognition,18(3), 569–577.
Dehaene, S., & Changeux, J. P. (2004). Neural mechanisms for access to consciousness. In M. Gazzaniga (Ed.), The cognitive neurosciences III (pp. 1145–1158). Cambridge, MA: MIT Press.
Dehaene, S., Changeux, J. P., Naccache, L., Sackur, J., & Sergent, C. (2006). Conscious, preconscious, and subliminal processing: A testable taxonomy. Trends in Cognitive Sciences, 100, 204–211.
Dennett, D. C. (1991).Consciousness explained. London: Penguin.
Di Lollo, V. (1980). Temporal integration in visual memory.Journal of Experimental Psychology: General,109, 75–97.
Dretske, F. (2004). Change blindness.Philosophical Studies,120, 1–18.
Dretske, F. (2006). Perception without awareness. In T. S. Gendler & J. Hawthorne (Eds.), Perceptual experience(pp. 147–180). Oxford: Oxford University Press.
Dretske, F. (2007). What change blindness teaches about consciousness.Philosophical Perspectives, 21, 215–230.
Fodor, J. A. (2007). The revenge of the given. In B. P. McLaughlin & J. Cohen (Eds.),Contemporary debates in philosophy of mind(pp. 105–116). Oxford: Blackwell.
Fodor, J. A. (2008). Preconceptual representation. In LOT2: The language of thought revisited (pp. 169–196). New York: Oxford University Press.
Friston, K. (2005). A theory of cortical responses.Philosophical Transactions of the Royal Society B, 360, 815–836.
Hollingworth, A., & Henderson, J. M. (1998). Does consistent scene context facilitate object perception?Journal of Experimental Psychology: General,127, 398–415.
Hollingworth, A., & Henderson, J. M. (1999). Object identification is isolated from scene semantic constraint: Evidence from object type and token discrimination. Acta Psychologica, 102, 319–343.
Hollingworth, A., & Henderson, J. M. (2002). Accurate visual memory for previously attended objects in natural scenes.Journal of Experimental Psychology,28, 113–136.
Irwin, D. E., & Yeomans, J. M. (1986). Sensory registration and informational persistence.Journal of Experimental Psychology: Human Perception and Performance,12, 343–360.
Jacob, P., & de Vignemont, F. (2010). Spatial coordinates and phenomenology in the two-visual systems model. In N. Gangopadhyay, M. Madary, & F. Spicer (Eds.),Perception, action and consciousness: Sensorimotor dynamics and two visual systems (pp. 125–144). Oxford: Oxford University Press.
Kouider, S., & Dupoux, E. (2004). Partial awareness creates the ‘‘illusion’’ of subliminal semantic priming.Psychological Science,15, 75–81.
Kouider, S., & Dupoux, E. (2007). How ‘semantic’ is response priming restricted to practiced items? A reply to Abrams & Grinspan (2007).Consciousness and Cognition,16, 954–956. Landman, R., Spekreijse, H., & Lamme, V. A. F. (2003). Large capacity storage of integrated objects
before change blindness.Vision Research,43, 149–164.
Liu, K., & Jiang, Y. (2005). Visual working memory for briefly presented scenes.Journal of Vision,5, 650–658.
Loftus, G. R., & Irwin, D. E. (1998). On the relations among different measures of visible and informational persistence.Cognitive Psychology,35, 135–199.
Luck, S. J., & Hollingworth, A. (2008).Visual memory. New York: Oxford University Press. Mewhort, D. J. K., Campbell, A. J., Marchetti, F. M., & Campbell, J. I. D. (1981). Identification,
localization, and ‘‘iconic memory’’: An evaluation of the bar-probe task.Memory & Cognition, 9, 50–67.
Noe¨, A. (2004).Action in perception. Cambridge, MA: MIT Press.
Oliva, A. (2005). The gist of a scene. In L. Itti, G. Rees, & J. K. Tsotsos (Eds.),The neurobiology of attention(pp. 251–256). London: Academic Press.
O’Regan, J. K., & Noe¨, A. (2001). A sensorimotor account of vision and visual consciousness. Behavioral and Brain Sciences,24, 939–1031.
Pelli, D. G., Palomares, M., & Majaj, N. J. (2004). Crowding is unlike ordinary masking: Distinguishing feature integration from detection.Journal of Vision,4, 1136–1169.
Pezdek, K., Whetstone, T., Reynolds, K., Askari, N., & Dougherty, T. (1989). Memory for real-world scenes: The role of consistency with scheme expectation. Journal of Experimental Psychology: Learning, Memory, and Cognition,15, 587–595.
Phillips, I. B. (in press). Perception and iconic memory.Mind & Language.
Roediger, H. L., & McDermott, K. B. (1995). Creating false memories: Remembering words not presented in lists. Journal of Experimental Psychology: Learning, Memory and Cognition, 21, 803–814.
Shanahan, M., & Baars, B. (2007). Global workspace theory emerges unscathed.Behavioral and Brain Sciences,30, 524–525.
Sligte, I. G., Scholte, H. S., & Lamme, V. A. F. (2008). Are there multiple visual short-term memory stores?PLoS ONE,3, 1–9.
Sperling, G. (1960). The information available in brief visual presentations. Psychological Monographs: General and Applied,74, 1–29.
Tarkiainen, A., Cornelissen, P. L., & Salmelin, R. (2002). Dynamics of visual feature analysis and object level processing in face versus letter-string perception.Brain,125, 1125–1136. Thorpe, S., Fize, D., & Marlot, C. (1996). Speed of processing in the human visual system.Nature,
381, 520–522.
Tye, M. (2006). Nonconceptual content, richness, and fineness of grain. In T. G. Szabo & J. Hawthorne (Eds.),Perceptual experience(pp. 504–530). Oxford: Oxford University Press. Tye, M. (2009). Consciousness revisited: Materialism without phenomenal concepts. Cambridge,
MA: MIT Press.
Van Gulick, R. (2007). What if phenomenal consciousness admits of degrees? Behavioral and Brain Sciences,30, 528–529.
Van Rullen, R., & Thorpe, S. (2001). The time course of visual processing: From early perception to decision making.Journal of Cognitive Neuroscience,13, 454–461.
Wertheim, A. H., Hooge, I. T. C., Krikke, K., & Johnson, A. (2006). How important is lateral masking in visual search?Experimental Brain Research,170, 387–402.
