the Current Tools
Background:Virtual reality (VR) is an evolving technology that has been applied in various aspects of medicine, including the treatment of phobia disorders, pain distraction interventions, surgical training, and medical education. These applications have served to demonstrate the various assets offered through the use of VR.Objective:To provide a background and rationale for the application of VR to neuropsy-chological assessment.Methods:A brief introduction to VR technology and a review of current ongoing neuropsychological research that integrates the use of this technology.Conclusions:VR offers numer-ous assets that may enhance current neuropsychological assessment protocols and address many of the limitations faced by our traditional methods. Key words:neuropsychological tests,technology,virtual reality
Maria T. Schultheis, PhD
Clinical Research Scientist Neuropsychology & Neuroscience
Laboratory
Kessler Medical Rehabilitation Research & Education Corporation
Department of Physical Medicine & Rehabilitation
University of Medicine & Dentistry
of New Jersey-New Jersey Medical School West Orange, New Jersey
Jessica Himelstein, MA
Postdoctoral Fellow
Department of Physical Medicine & Rehabilitation
University of Medicine & Dentistry of New Jersey-New Jersey Medical School West Orange, New Jersey
Albert A. Rizzo, PhD
Assistant Research Professor Integrated Media Systems Center School of Gerontology
University of Southern California Los Angeles
I
N THE PAST few years, virtual reality (VR) has undergone a transition that has taken it out of the realm of expensive toy and into that of functional technology. Continuing ad-vances in VR technology, along with concomi-tant system cost reductions, have supported the development of more usable, useful, and accessible VR systems. These systems have the potential to uniquely target a wide range of physical, cognitive, and behavioral human issues and research questions. Neuropsychol-ogy is one discipline that may clearly ben-efit from the application of VR technology.Address correspondence and requests for reprints to Maria T. Schultheis, PhD, Neuropsychology and Neuroscience Laboratory, Kessler Medical Rehabil-itation Research & Education Corporation, 1199 Pleasant Valley Way, West Orange, NJ 07052. Tele-phone: (973) 324–3528; Fax: (973)243–6984. E-mail: mschultheis@kmrrec.org.
Supported in part by grant #HD08589–01 from the Na-tional Institute of Child Health and Human Develop-ment, grant #H133G000073 from the National Insti-tute on Disability and Rehabilitation Research, and by the Integrated Media Systems Center, a National Science Foundation Engineering Research Center, Cooperative Agreement No. EEC-9529152.
J Head Trauma Rehabil2002;17(5):378–394
c
°2002 Lippincott Williams & Wilkins, Inc. 378
The application of VR in neuropsychology is so distinctively important because it rep-resents more than a simple linear extension of existing computer technology for human use. VR offers the potential to deliver system-atic human testing and training environments that allow for the precise control of com-plex dynamic three-dimensional (3D) stimu-lus presentations, where sophisticated behav-ioral recording is possible. When combining these assets within the context of function-ally relevant, ecologicfunction-ally valid virtual environ-ments, a fundamental advancement emerges in how human behavior can be addressed in many scientific disciplines.
In the broadest sense neuropsychology can be defined as an applied science that evalu-ates how specific activities in the brain are expressed in observable behaviors.1Effective neuropsychological assessment is often a key component in both the treatment and the sci-entific analysis of many central nervous sys-tem disorders and can serve a number of func-tions. These include the use of psychometric evaluation tools for diagnostic purposes, the provision of normative data for comparison with impaired cognitive and functional abil-ities, the specification of cognitive strengths and weaknesses to help inform the design of rehabilitative strategies, and provision of metrics to assess treatment efficacy or plot neurodegenerative decline. Neuropsycholog-ical assessment also serves to create data for the scientific understanding of brain function-ing through the examination of measurable se-quelae that occur after brain damage or dys-function. To date, most neuropsychological research has focused on increasing our under-standing of component cognitive processes, such as attention, executive functions, mem-ory, and language spatial abilities. More re-cently, understanding the role of cognitive processes during functionally relevant tasks (i.e., instrumental activities of daily living) has evolved as an area of focus within
neuropsy-chological research. VR offers the potential to integrate these two aspects of neuropsychol-ogy and provide an innovative approach to un-derstanding brain–behavior relationships.
In view of the promising benefits that VR can offer neuropsychological assessment, treatment, and research, this article seeks to (1) provide a brief introduction to VR tech-nology and the potential benefits of the ap-plication of VR to the field of neuropsychol-ogy, (2) provide a brief overview of recent studies that have successfully applied VR to the assessment of component cognitive abili-ties and functional evaluations, and (3) discuss important considerations in the application of VR.
WHAT IS VR TECHNOLOGY?
VR can be viewed as an advanced form of human–computer interface that allows the user to “interact” with and become “im-mersed” in a computer-generated environ-ment in a naturalistic fashion.2 By analogy, much like an aircraft simulator serves to test and train piloting ability, computer-generated interactive virtual environments (VEs) can be designed to assess and rehabilitate cognitive and functional abilities by means of expo-sure to simulated “real world” and/or analog tasks. Interaction in three dimensions is a key characteristic that distinguishes a VR expe-rience from other technologies. The believ-ability of the virtual experience (or “sense of presence”) is fostered by the use of such specialized technology as head-mounted dis-plays (HMDs), tracking systems, earphones, gesture-sensing gloves, and haptic-feedback devices. An HMD is an image display system worn on the head (like a diving mask) that remains optically coupled to the user’s eyes as he or she turns and moves. A tracking sys-tem senses the position and orientation of the user’s head (and HMD) and reports that in-formation to a computer that updates (in real
Fig 1.Person in a virtual environment wearing an HMD and head tracker.
time) the images for display in the HMD. In most cases full-color stereo image pairs are produced, and earphones may deliver rele-vant 3D sound. The combination of an HMD and tracking system allows the computer to generate images and sounds in any computer-modeled (virtual) scene that corresponds to what the user would see and hear from their current position if the scene were real (Figure 1). The user may walk and turn around to survey a virtual landscape or inspect a vir-tual object by moving toward it and peer-ing around its sides or back. Although HMDs are most commonly associated with VR, other methods incorporating 3D projection walls and rooms (known as CAVES), as well as basic flat-screen computer systems, have been used to create interactive scenarios. Methods for navigation and interaction such as data gloves, joysticks, and some high-end “force feedback” mechanisms that can provide haptic feedback
are also available and can be used to further enhance realism and the “suspension of dis-belief” required to generate the sense of pres-ence within a VR environment.
Although VR remains a developing tech-nology, it is not a new addition to the medi-cal field. Various studies have served to docu-ment the successful integration of VR to myr-iad aspects of medicine, including surgical training,3 education of patients and medical students,4and treatment of sensory mobility deficits.5Treatment of psychological dysfunc-tion, including anxiety disorders and phobia, has been successfully reported by numerous researchers who have applied the use of VR for treating fear of flying and fear of heights.6,7 VR has also been applied to the treatment of posttraumatic stress disorder (PTSD),8 eat-ing disorders,9 and pain management.10 Re-sults from these studies have served to un-derscore some of the specific assets afforded
through VR applications and advantages over other more traditional assessment protocols. WHAT ARE THE ADVANTAGES OF APPLYING VR?
Earlier discussions of VR technology in other application areas suggest a number of assets for its use.11,12 Among these are sev-eral advantages that are of direct relevance to neuropsychological research, assessment, and treatment.
One key benefit of applying VR protocols in the assessment of cognitive skills and abili-ties is its more naturalistic or “real-life” testing environment, which may serve to address one of the most commonly noted limitations of tra-ditional neuropsychological measures.13That is, VEs may be especially well suited to im-proveecologicalvalidity, or the degree of rel-evance or similarity that a test has relative to the “real” world. The complexity of stimulus challenges found in naturalistic settings could be delivered while still maintaining the ex-perimental control required for rigorous sci-entific analysis. Results would have greater clinical relevance and could have direct impli-cations for the development of more effective functional rehabilitation approaches.
Full control over stimulus presentation and response measurement is another asset of-fered through the use of VR technology. Clin-icians or researchers can specify the cre-ation of highly controllable VR environments. Specifically, the use of VR protocols permits such factors as the number, speed, and/or or-der of stimulus presentation to be effortlessly manipulated while maintaining an objective
means of data collection on relevant target re-sponses. In addition, the relative flexibility of stimulus presentation in a VE can allow grad-ual increments of difficulty and challenge to be presented to the individual. This could al-low for individualization of assessment and treatment, while still maintaining consistency in desired outcome measures. As such, VR
technology could be used to supplement ex-isting neuropsychological assessment proce-dures that traditionally rely mainly on pencil-and-paper tests and behavioral observation, potentially leading to improvements in psy-chometric reliability and validity and promot-ing the independent replication of research findings needed for scientific progress in this field.
Another unique asset of VR for neuropsy-chological assessment is the opportunity to evaluate cognitive functions withindynamic
interactions and environments. That is, al-though traditional measures can provide infor-mation regarding component cognitive func-tions and potentially predict how deficits may translate into the “real” world, VR offers a medium to examine this relationship directly and allow for the evaluation of complex cog-nitive behaviors. For example, although tradi-tional measures may separately assess differ-ent systems of memory functioning (i.e., vi-sual memory, verbal memory) within a VE, clinicians and researchers can evaluate how these two systems may or may not be used concomitantly by individuals during “real-life” tasks. Furthermore, such evaluations could serve to better identify the particular compen-satory strategies initiated by individuals dur-ing these tasks, potentially resultdur-ing in more applicable and usable neuropsychological re-commendations.
In summary, the various assets offered by VR, including increased ecological validity, objectivity, and assessment in interactive and functionally relevant scenarios could support a number of opportunities for applications in both the clinical and research domains of neuropsychology. In recent years, a growing number of researchers have begun explor-ing the use of VR technology for applications designed to target neuropsychological assess-ment with populations having central nervous system dysfunction. Although the breadth of this clinical literature pales by comparison with VR research in the testing and training
area with normal populations, the initial ef-forts using VR designed for impaired clinical groups are encouraging. Summaries of some relevant studies that have applied VR to the evaluation of specific cognitive functions are provided in the following.
COGNITIVE FUNCTIONS Executive functions
Alexander Luria14 defined executive func-tioning as those functions that are involved in the planning, regulation, and verification of an action. This definition has been ex-panded to include “a set of behavioral compe-tencies which include planning, sequencing, the ability to sustain attention, resistance to interference, utilization of feedback, the abil-ity to coordinate simultaneous activabil-ity, cog-nitive flexibility and the ability to deal with novelty.”15 Given the complexity of this cog-nitive construct, the use of traditional psy-chometric methods for the assessment and re-habilitation of executive functions has been questioned.16,17
One group of researchers has developed an HMD VE system specifically designed for the assessment and rehabilitation of this pro-cess in persons with traumatic brain injury (TBI), multiple sclerosis (MS), and stroke.17,18 Using a standard tool of neuropsychological assessment as a model (Wisconsin Card Sort-ing Test—WCST), these researchers have cre-ated a virtual building that requires the person to use environmental clues in the selection of appropriate choices (doorways) to navigate from room to room in the structure. The door-way choices can vary according to the cate-gories of shape, color, and number of port-holes, and the person is required to refer to the previous doorway for clues as to the ap-propriate next choice. When the administra-tor surreptitiously changes the choice crite-rion, the person being tested is then required to shift cognitive set, analyze clues, and
de-vise and initiate a new choice strategy. The researchers concluded that the cognitive re-quirements of this task were similar to those included within the WCST.
Using this VE system, Pugnetti et al19 com-pared a mixed group of neurological patients (MS, stroke, and TBI) with healthy controls on performance on both the WCST and the VE executive function system. The results of this study were concordant with previ-ous anecdotal observations by family mem-bers regarding the patient’s everyday behav-ior. Although the psychometric properties of the VE task were comparable to the WCST in terms of gross differentiation of patients and controls, weak correlations between the two methods suggested that they were mea-suring different functions. In this regard, the VE task was seen to specify impairments ear-lier in the test sequence compared with the analog pencil-and-paper test. The authors sug-gest that “. . .this finding depends on the more complex (and complete) cognitive demands of the VE setting at the beginning of the test when perceptuomotor, visuospatial (ori-entation), memory, and conceptual aspects of the task need to be fully integrated into an efficient routine.”19p.160 The detection of these “integrative” difficulties is of vital im-portance when predicting real-world capabil-ities from test results. The results of this study also included a discriminant function analy-sis that suggested the VR test was more effec-tive at correctly classifying groups of healthy controls and patients with neurological dam-age than the WCST. Further support for the enhanced ecological validity of VR assess-ment can be seen in a detailed single-subject case study of a stroke patient using this sys-tem. In this report,20 results indicated that the VE was more accurate than the WCST in specifying executive function deficits in a highly educated patient 2 years after a stroke. The VE was successful in detecting deficits that had been reported to be limiting the patient’s everyday performance yet were
missed using traditional neuropsychological tests.
Most recently, Elkind et al21 devised a flat-screen VR analog of the WCST, similar in con-cept to the design of Pugnetti et al,22but with enhanced graphics and a more detailed envi-ronment. Although Pugnetti et al’s scenario in-volved navigation through a house, the Elkind et al abstract reasoning scenario consisted of a beach scene, including four beach umbrel-las each containing one of four stimuli. The four stimuli consisted of common objects dif-fering in color and number. For the task, the participant was presented with a new object and was required to match the object to one of the four stimuli. Feedback regarding the ac-curacy of match performance was provided with each new item. To complete the test successfully, the participant had to learn the matching principle being used to determine the correct stimuli choice and use that princi-ple to guide future choices. Correlational anal-ysis of healthy control performance on Elkind et al’s VR analog and the WCST indicated that participants generalized the strategies learned through administration of one measure and applied these techniques to completion of the second measure. Excluding one score on the WCST (perseverative errors), all performance scales on the WCST and VR analog were di-rectly related, with correlations ranging from
r=.33 (P<.01) to r=.58 (P<.001). The investigators interpreted this finding as evi-dence of learning transfer between the WCST and the VE. Although the VR task was more difficult for the participants than the WCST, subjective report indicated higher motivation levels for the VR task. Thus, use of the VR analog seems to alleviate the issues of poor motivation and ceiling effects present during paper-and-pencil cognitive testing.
McGeorge et al23 addressed the planning component of executive functioning. The au-thors observed the drawbacks of using tradi-tional neuropsychological tests as measures of
planning, including the lack of long-term con-tinuous planning requirements, the absence of competing subtasks, and the poor relation-ship to life situations. Limitations of real-world assessments of planning skills include safety concerns and an inability to control for environmental factors that may confound per-formance. Rationale for the use of a VE in-cluded alleviation of most safety concerns, minimization of restrictions present in the real world because of physical impairments, the manipulation or elimination of environ-mental distracters, and improved relevance to real-life situations. Specifically, McGeorge et al’s23 work involved participants with TBI who were reported by family members to have planning deficits. Their performance was compared with that of healthy controls on both virtual and actual vocationally ori-ented planning tasks. The tasks were adminis-tered within a simulated work setting, where participants were given 20 minutes to com-plete a maximum number of work-related errands. Examples of errands included collect-ing office equipment from a cupboard, read-ing materials, addressread-ing envelopes. TBI par-ticipants performed significantly worse than the healthy controls in both environments. The authors concluded that support for the enhanced ecological validity of VR assess-ment of planning skills was provided by the high correlation between number of errands completed in the real-world and virtual envi-ronments (r=.79, P<.01). The three afore-mentioned VEs seem to offer advantages for the valid and reliable assessment of execu-tive functions relaexecu-tive to the more traditional paper-and-pencil tests typically used by neu-ropsychologists.
Attention processes
Sohlberg and Mateer24 presented a clini-cal model of attention that outlines various levels of this basic cognitive process, includ-ing focused attention, sustained attention,
selective attention, alternating attention, and divided attention. The specific asset offered by VE technology for the assessment of at-tentional subprocesses is the provision of a controlled stimulus environment in which cognitive challenges can be presented along with the precise delivery and control of “dis-tracting” auditory and visual stimuli. Efforts to assess attention are supported by the widespread occurrence of attentional deficits in numerous conditions across the lifespan, including attention deficit hyperactivity dis-order (ADHD), acquired brain injury (ABI), and as a feature of various neurodegenerative disorders (e.g., Alzheimer’s disease, vascular dementia). Although some investigators have highlighted the advantages that VR offers for the assessment of attention functions,25,26few studies have been conducted using VR to in-vestigate this cognitive process.
One group, Rizzo et al,27has designed a VR classroom environment to begin to examine the different aspects of attention in children with ADHD. The “Virtual Classroom” is an HMD VR system that consists of a standard rectangular classroom environment contain-ing desks, a female teacher, a blackboard across the front wall, a side wall with a large window looking out onto a playground, and a street with moving vehicles, and, on each end of the opposite wall, a pair of doorways through which activity occurs. Within this scenario, children’s attention is assessed while a series of typical classroom distracters (e.g., ambient classroom noise, activity occur-ring outside the window) are systematically controlled and manipulated within the VE. The child sits at a virtual desk within the vir-tual classroom, and attention can be measured in terms of reaction time performance and er-ror profiles on a variety of attention challenge tasks that are delivered visually using the bla-ckboard or auditorily by the teacher’s voice.
In the user-centered design phase, 20 nondiagnosed children (ages 6–12) were
tested on basic selective and alternating attention tasks. Feedback pertaining to aesthetics and usability of the VE was so-licited and incorporated into the iterative design-evaluate-redesign cycle. Overall results indicated little difficulty in adapting to use of the HMD, no self-reported occurrence of side effects determined by posttest interviews using the Simulator Sickness Questionnaire,28 and excellent performance on the stimulus tracking challenges. After this phase, a clin-ical trial that compared 9 physician-referred ADHD boys (age 6–12) with 10 nondiagnosed children. The attention testing involved a continuous performance task delivered on the blackboard that required the participants to hit a response button whenever they saw the letter “X” preceded by the letter “A.” Two 10-minute conditions were presented to participants: one without distraction and one with distractions (pure audio, pure visual, and mixed audiovisual). VR performance was compared with results from standard neu-ropsychological testing. Preliminary findings from this study revealed significant difference in omission and commission error perfor-mance between ADHD children and nondiag-nosed children, with the children with ADHD making more omission errors in the distract-ing condition. In addition, the number of movements made was also found to be signifi-cantly greater in children with ADHD relative to nondiagnosed children (tracked from head, arm, and leg). These initial findings suggest that performance in the “Virtual Classroom” could serve to help discriminate and diagnose the symptoms of ADHD among children. The “VR Classroom” may have potential as an efficient, cost-effective, and scalable tool for measuring attention performance beyond what exists using traditional methods.
Investigation into performance on VE di-vided attention tasks by individuals with TBI and healthy controls has been conducted by Lengenfelder et al.29 The VE consisted of a
driving course, where the primary task was to “drive a car” and the secondary task re-quired the correct identification of a four-digit number presented during the driving course. The numbers were presented in a varied lo-cation (always in same place versus random placement) on the “windshield” of the “vehi-cle”(i.e., computer screen) and varied presen-tation rate (2.4 versus 0.6 ms) to differentiate between simple and complex divided atten-tion. The simple attention condition required the subject to “drive” through the course (pri-mary task) and identify the four-digit num-ber (secondary task) that repeatedly appeared in the center of the screen at a slow rate. By contrast, the complex condition required the subject to “drive” through the course (pri-mary task) and identify the four-digit number (secondary task) that appeared randomly on screen at a very fast rate. Performance mea-sures included driving speed and correct num-ber of stimuli identification. Pilot data found no differences in speed management, across both simple and complex tasks, between the two groups and reported that both groups drove faster when stimuli presentation was at a faster rate. The findings also suggested that the participants with a TBI had greater difficulty completing the secondary task than healthy controls. Early evidence for the valid-ity of this VE as a measure of divided atten-tion was provided by means of the correlaatten-tion between performance on the VR task and on traditional neuropsychological measures of di-vided attention. Spearman’s correlations cal-culated between the VR performance index of divided attention and neuropsychological measures of divided attention yielded corre-lations that ranged from r= −.40 (P =.44) tor= −.94 (P=.01). These correlations indi-cated that the more errors that were made on the VR divided attention task, the lower the number of correct responses on neuropsycho-logical measures of divided attention. Thus the “Virtual Classroom” of Rizzo et al27 and
the driving course of Lengenfelder et al29 pro-vide two viable options for the use of VR to assess attentional functions in varied clinical and healthy populations.
Visuospatial processes
Spatial ability is commonly broken down and defined in terms of the tests used to mea-sure its various subcomponents (e.g., spatial perception, orientation, visualization, men-tal rotation). However, Carpenter and Just30 present a general definition of this process as the ability to generate “a mental represen-tation of a two or three-dimensional struc-ture and then assessing its properties or per-forming a transformation of the representa-tion”(p. 221). VR has been applied to examine spatial processes and their generalization to everyday functions. For example, flat-screen VE systems have been used in studies of hu-man “place learning” ability.31–34 Flat-screen, joystick-controlled VEs modeled after the Mor-ris Water Maze task35 have been designed to allow a person to navigate an environment in search of a hidden platform. Studies using this VE have reported significant effects in the areas of aging, CNS dysfunction, and gender differences.31,34 One recent study examining performance of this task in individuals with TBI demonstrated that these individuals have significantly impaired VE place learning abil-ities, which correlated with self-reported fre-quency of wayfinding problems in everyday life.36
Other flat-screen VE scenarios have also shown promising results for training spatial orientation and navigation skills with phys-ically disabled children whose inability to
independentlyexplore environments may en-gender developmental impairments in cogni-tive mapping abilities (i.e., the ability to visu-alize a route in the mind’s eye).37In a series of four studies reviewed in Stanton et al,37 chil-dren with physical disabilities were allowed to independently explore VEs modeled after
school environments. Transfer of spatial learn-ing was shown to generalize to the actual school that was modeled, and this learning improved with practice. Also, flexibility in the cognitive representation, or “mapping,” of the environment was inferred, because the children were able to accurately indicate the direction of objects that were not in their line of sight. Another group that developed a visuospatial VE scenario has reported sim-ilar findings.38 Successful learning and trans-fer of object locations were found using a flat-screen “classroom” VE with unimpaired chil-dren. In this study the spatial performance equaled “real-world” training, and successful transfer to the real environment was reported. Similar spatial learning using a flat-screen VE training system targeting functional spatial knowledge has also been reported for adoles-cents with developmental disabilities in a su-permarket search-and-navigation scenario.39
Taking a more component-based approach to addressing spatial ability, Rizzo et al40 have developed a suite of projection-based 3D holographic VR-delivered applications targeting mental rotation, depth percep-tion, manual movement stability, 3D field dependence (3D rod and frame test), 3D manual tracking, and visual field–specific re-action time. An initial study with this system specifically targeted mental rotation (MR), a dynamic imagery process that involves “turning something over in one’s mind.”41In the VE system, MR was assessed by means of a manual spatial rotation task that required subjects to manipulate virtual block configu-rations. Subjects were presented with a target block configuration, and the speed and effi-ciency of their movements to superimpose a replica design on the target was measured and recorded. An initial feasibility study with adult healthy controls investigated self-reported side effect occurrence, learning on the VE MR task, transfer of training from the VE to a pencil-and-paper MR test,42associations
be-tween VE performance and other “standard” tests of cognitive performance, and gender differences. A number of encouraging find-ings emerged, including minimal side effect occurrence, good psychometric properties of the VE test, provocative relationships with standard neuropsychological tests, a lack of gender differences compared with the pencil-and-paper measures, training improvement, and significant transfer of training.27The full VE system (all visuospatial scenarios cited earlier) is currently being tested with normal elderly subjects and comparisons of these data with central nervous system–impaired populations (Alzheimer’s disease, TBI) is planned. Given the myriad number of abili-ties included under the rubric of visuospatial functions, as well as the intuitive match between VR and assessment of visuospatial functions, more research has been conducted in this area relative to the use of VR for assessment of other cognitive functions. Memory functions
The assessment and rehabilitation of mem-ory disorders has received considerable at-tention in the neuropsychological literature. This has resulted in the better specification of memory processes, explanation of underly-ing central nervous system mechanisms, and a greater effort to determine the problems that occur in everyday life because of clin-ical memory disorders. This has also led to the acceptance of memory as a multicom-ponent system, including sensory memory, short-term memory, and long-term memory,43 and working memory.44VR technology could be of value for improving the specificity of memory assessment for more effective diagnosis.
Targeting of memory processes in VEs is closely related to VR work in the area of “spatial orientation and way-finding.” To this end, research examining memory for objects and spatial layout contained in a VE has been
underway since the mid-1990s at the Univer-sity of East London (UEL). The UEL group has focused on specifying the types of mem-ory that may be enhanced during a four-room house navigation task.45This approach uses a flat-screen system with a joystick in-terface, which allows one subject to navi-gate the house (active condition), whereas a yoked subject is simply exposed to the same journey but has no navigational control (pas-sive condition). Both subjects are directed to seek out an object (toy car) during the explo-ration, and differential memory performance between the two groups on spatial versus ob-ject memory of the environment is tested. In initial tests with normal populations, it was observed that the active groups showed bet-terspatialmemory for the route, whereas the passive group displayed superiorobjectrecall and recognition memory for the items viewed along the route.46 Although one group repli-cated these results,22 others have reported mixed findings.13,47
Another study using this VE with clini-cal populations (MS and stroke) has pro-duced results that suggest the value of this type of VE application to inform neuropsy-chological assessment. That is, performance was significantly poorer among stroke partic-ipants relative to performance of healthy con-trols. Interestingly, Although the typical spa-tial/content memory dissociation was found with unimpaired groups (i.e., active=better spatial memory; passive=better object mem-ory) and the active stroke group displayed bet-ter spatial memory compared with the pas-sive group, the stroke patients displayed no advantage on object memory while in the pas-sive condition.45 Similar findings using the UEL scenario were also reported by Pugnetti et al19 in a study comparing individuals with MS to healthy controls. Another example of the use of VEs for spatial memory assessment was provided in a case report of a severely
amnestic stroke patient who showed signifi-cant improvements in her ability to find her way around a rehabilitation unit after train-ing within a VE modeled after the unit.48,49To date, investigations of memory functions us-ing VEs have focussed far on spatial and object memory, and given the positive results future research using VR to address other aspects of memory functions may be fruitful.
INSTRUMENTAL ACTIVITIES OF DAILY LIVING (IADLs)
Some of the VE systems targeting compo-nent cognitive processes reported on previ-ously could conceivably be reviewed here (i.e., virtual classroom, spatial navigation). The primary focus of this section is behav-iors required for functional activities and/or IADLs. These functional approaches empha-size the design of ecologically valid VEs to test and train more integrated behavioral reper-toires in populations who are limited in their capacity for real-world independent learning. Advances in this area could be of consider-able value compared with the costly and time-consuming efforts that are required for the in vivo, one-on-one approaches often required with certain clinical populations. In this way, VEs can be used to assess functional behavior and deliver consistent, hierarchical, and safe training that could free the therapist’s time to be spent in other more necessary and inten-sive one-on-one client contact.
Navigational skills
VEs targeting functional skills have been developed in the areas of street crossing, wheelchair navigation, meal preparation, use of public transportation, driving, and obsta-cle avoidance. Strickland50reported initial re-sults on the feasibility of using an HMD with autistic children with the long-term goal of targeting VE training for safe street crossing.
The two autistic children initially reported on adapted to the headset and were able to track moving automobile stimuli and select objects. Continued efforts with this population51and with children having motor disabilities52 are underway to target the functional behaviors required for safe street crossing in a VE. In-man et al51have also reported success in train-ing children with motor impairments to use motorized wheelchairs in a variety of simu-lated real-world conditions using an HMD VE system.
Cooking behaviors
Food preparation skills within a virtual kitchen scenario have also been investigated. Christiansen et al53 have developed an HMD VE of a kitchen in which persons with TBI have been assessed in terms of their ability to perform 30 discrete steps required to prepare a can of soup. Contingent on success at each step, various auditory and visual cues can be presented to help prompt successful perfor-mance and learning. In the first reported pi-lot study with this system, 30 patients with closed-head injuries were able to successfully function using the HMD with minimal side ef-fects. In addition, acceptable reliability coef-ficients were reported using a test/retest as-sessment. The stability of the VR assessment scores was analyzed using the intraclass corre-lation approach, coefficients of variation. The coefficient of variation for method error was computed for all intraclass correlations and ranged from 1.86% to 14.32%, indicating that the stability of most test items was good and that any low correlations were reflective of re-duced variability and not a lack of consistency between test and retest. These researchers also report ongoing enhancements to the sys-tem regarding the delivery of more complex challenges and increased flexibility in the pre-sentation of cueing stimuli.
Learning and memory assessment in vocational environment
One project, which is currently examin-ing the use of VR for evaluatexamin-ing vocation-ally relevant cognitive functions, is being con-ducted with the use of a “Virtual Office” scenario.54Specifically, the VR Office is a rel-atively generic office scenario in which the subject could be seated at a desk or could be standing up. Currently on the subject’s desk are a computer, telephone, and various desk accessories. There are other desks in the room, as well as filing cabinets and wall deco-rations. In addition, the office contains a large picture window and a clock that marks real time. A first study, examining learning and memory in individuals with TBI and MS, will use 16 target items (8 common and 8 uncom-mon) that will be placed throughout the VR Office. Measures on learning trials and ver-bal and visual recall will be collected. It is anticipated that learning and memory within the VR Office will allow for the use of verbal and visual memory compensatory strategies. As such, direct evaluation of performance may help to identify self-initiation of strategy use and allow evaluation of effectiveness of strategies. It is anticipated that such obser-vations will allow for improvements in the identification and training of compensatory strategies.
Driving assessment
Virtual environments that target driving ability have been tested in TBI and elderly populations. Liu et al55reported that an HMD driving scenario successfully discriminated between the TBI group and an unimpaired group and that age effects were detected with this system. More recently, researchers have examined the use of VR Driving Assess-ment System for determining driving capac-ity after acquired brain injury (stroke, TBI).56
The study compares a VR-based driving assessment (VRDS) protocol with the cur-rent “gold standard” of driving assessment, the “behind-the-wheel evaluation.” Specifi-cally, the VRDS program consists of three driv-ing environment scenarios. The first scenario was selected to allow direct comparison be-tween driving performance on the behind-the-wheel (BTW) evaluation and a VR analog of this BTW route. The additional two driv-ing scenarios were selected to allow exami-nation of challenging factors on driving abil-ity, including the addition of nighttime driving and “stressful” driving (i.e., congested traffic, pedestrian and emergency situations). Given the scarcity of studies with clinical popula-tions in these environments, the effects of par-ticipation in a VRDS are also under investi-gation. That is, factors such as incidence of side effects, usability of the technology, and evaluation of factors contributing to perfor-mance (i.e., computer experience) are being examined. The pilot phase of the study is cur-rently underway, and early observations sug-gest good usability of the technology with this clinical population.
Finally, one particularly noteworthy and ambitious effort to address functional life skills is the work of the University of Not-tingham group in the development of the “Virtual City.”57 This flat-screen scenario ad-dresses independent living skills in persons with learning and developmental disabilities, and the researchers have implemented good user-centered design principles by incorporat-ing user group input in the selection of envi-ronments targeted in the application. These settings were designed to address the use of public transport, road safety, home safety, use of public facilities within a caf´e, and shop-ping skills in a large supermarket. An initial study tested 20 subjects with mental retarda-tion and addressed usability, enjoyment, skill learning, and transfer of skills to the real en-vironment using subjects’ self-report and
be-havioral observation methods.58 Subjects re-ported high enjoyment of the task and more “ease of use” with the system than antici-pated in the “expected” usability ratings de-termined by support workers. Transfer from the VE to the real environment for some skills was noted in this initial feasibility study. Work with this evolving scenario continues, and re-sults from more systematic measurement of learning and transfer are anticipated. Such re-sults are important to support previous ob-servations of transfer seen with these popu-lations. For example, Cromby et al39reported evidence of positive learning transfer for an-other group of developmentally disabled stu-dents using a supermarket VE modeled af-ter an actual market. Subjects were reported to be better able to navigate within and se-lect specific items in the real supermarket after training in the VE, and these subjects outperformed those who practiced in non-specific VEs. Positive results were also re-ported for a public transit VE training program called “Train to Travel.”59 This HMD VE sys-tem was developed in the mid-1990s to as-sess and teach persons with developmental disabilities how to use key routes on the Mi-ami Valley Regional Transit System. However, because the VE component was part of a com-prehensive training package, including video and multimedia tutorials, the specific effect of the VE on successful independent transit use was not determinable. Future studies that methodologically target the specific effects of the VE “ingredient” are necessary to deter-mine whether these types of programs are ef-fective for efficiently and economically target-ing functional abilities. The aforementioned investigations represent essential “first steps” in determining whether VE training can fos-ter transfer of learning to activities of daily liv-ing. For persons whose learning abilities are challenged because of central nervous system dysfunction, this line of research is especially important.
CONCLUSIONS
Although still an evolving technology, VR seems to offer many opportunities for the de-velopment of innovative, cost-effective meth-ods of assessment and treatment of a variety of neuropsychological impairments. Al-though initial findings may be provocative, analysis of the cost-benefit issues for the application of VR technology is needed to better determine where VR may optimize neuropsychological research, assessment, and treatment. A detailed discussion of these topics can be found in Schultheis and Rizzo,12 but some important considerations are highlighted in the following.
While considering the advantages offered by VR, it is important to mention the mini-mal risk of side effects that is present when using VR technology. The most common risk of participation in a VE task is simulator sick-ness. Simulator sickness and motion sickness are similar in symptom constellation, but pre-sentation is less severe in simulator sickness. Possible symptoms include vertigo, dizziness, headaches, and sweating. Although the rate of simulator sickness in clinical populations is not well known, prior studies reviewed herein have reported minimal (<10%) nega-tive reactions in participants with MS and TBI. Additional research to further explain the im-pact of simulator sickness and/or other risks for VR participation of clinical populations will be crucial to the development and appli-cation of this technology.
Another consideration involves questions frequently asked regarding the cost and avail-ability of VR technology.60–63 Continued ad-vances in technology (i.e., hardware and soft-ware evolvement) and the steady decrease in prices of VR systems have served to minimize these concerns. A further concern has been raised regarding VR’s ability to fit the needs of clinical populations. That is, although much of the work incorporating VR approaches has
included healthy individuals and some psychi-atric populations, studies that have included populations with neurological impairments (e.g., TBI, dementia) remain sparse. However, as previously noted, the few studies that have used clinical subject groups have reported a low rate of the most commonly seen adverse side effect of VR use (i.e., simulator sickness). Given the minimal risk level, the cost-benefit ratio for the use of VR as a research tool in clin-ical populations is favorable. Other consid-erations, including the relationship between disability (i.e., cognitive or physical impair-ment) and the complexity of VR systems, will require further research when considering future clinical applications. For example, the development of a VR environment for assess-ment purposes may be significantly altered by the selection of appropriate hardware (i.e., HMD versus flat screen). Although the en-hanced 3D nature of the HMD and CAVE VR systems may provide more ecological validity than the flat-screen VR systems, the external “real-world” feedback that is provided when using flat-screen environments may minimize the adverse side effects possible with HMD and CAVE VR systems. However, the sense of immersion reported by participants in HMD, CAVE, and flat-screen VR environments has yet to be systematically compared.
Research comparing the use of similar scenarios in HMD, CAVE, and flat-screen VR systems across healthy and clinical popula-tions would also provide valuable information regarding the most appropriate environment for assessment purposes. Further research is also called for to test the notion that VEs are inherently ecologically valid. This may be accomplished by testing the same cognitive function across different VR scenarios. If VR scenarios do indeed tap the reported cogni-tive functions, then similar patterns of strengths and weaknesses should be revealed independent of the VR scenario used. The selection of which tasks (i.e., cognitive,
physical, functional) may be best addressed using VR with patient populations is an area that will require considerable future research. Although some applications may seem to be a “natural” match (i.e., driving), careful development of future applications should maintain a focus on the user and be driven by specific identified needs.
Finally, ethical considerations will need to be clarified as new applications of this tech-nology continue to evolve. For example, al-though VEs may be developed for clinically justifiable purposes (e.g., phobia treatment), it must be considered that the control of the VR experiences presented to the patient car-ries with it the capacity (and responsibility) to influence personal development of those patients exposed to this control.64 Along these same lines, a number of authors have acknowledged the potential difficulties that could arise with the use of VEs by individuals with certain types of psychopathology.65–67 An additional ethical concern raised by VR research is the mandate that research partic-ipants not be deceived. Given the “sense of immersion” and “presence” created in VR re-search relative to other computerized labo-ratory measures, participants may be more likely to believe that they are actually ex-periencing real-world events. Thus, events that transpire within VEs may be particu-larly salient for participants. This could be addressed by a debriefing session at the completion of the assessment session. Fur-thermore, the supposed enhanced ecological validity of VR may lead examiners to over-estimate the relevance of test results to ac-tual functional behaviors. A comprehensive discussion regarding the various ethical
con-siderations in the application of VR technol-ogy and clinical treatment and research can be found in Rizzo et al.40
In summary, like many emerging computer and information technologies, VR potentially offers numerous advantages and solutions to the field of neuropsychology. Clearly, an affordable tool that offers consistent yet mod-ifiable ecologically valid testing and training environments tailored to the individual’s needs could provide significant benefits to individuals with disabilities. The technology (for both hardware and software) continues to move forward, and work with clinical popula-tions remains to be done. Although it is clear that a fair amount of work has been initiated in applying VR for the assessment of cogni-tive functions in clinical populations, much work is still needed to fully determine VR’s contribution to neuropsychology. As such, the field is at a turning point, where much of the nascent efforts that hint at the potential of this technology need to be further explored and developed in depth. Effective application of VR will require the collaborative efforts of rehabilitation specialists, computer scientists, and engineers, and it is only through the integrated expertise of multiple fields that the potential benefits of VR can be fully realized. Although numerous challenges remain, the potential to augment traditional neuropsy-chological testing is impressive, and this application domain is at the early stages of development. As a larger body of VR-related literature emerges, we will be in a better po-sition to formulate the standards for research-based and clinically oriented applications. This could result in a new paradigm for using VR in research and clinical neuropsychology.
REFERENCES
1. Lezak MD.Neuropsychological Assessment, 3rd ed. New York: Oxford University Press; 1995.
2. Rizzo AA, Buckwalter JG. Virtual reality and cogni-tive assessment and rehabilitation: the state of the
art. In: Riva G, ed.Psycho-neuro-physiological As-sessment and Rehabilitation in Virtual Environ-ments: Cognitive, Clinical and Human Factors in Advanced Human Computer Interactions. Amster-dam: IOS Press; 1997:123–146.
3. Cosman PH, Cregan PC, Martin CJ, Cartmill JA. Vir-tual reality simulators: current status in acquisition and assessment of surgical skills.ANZ J Surg. 2002; 72(1):30–34.
4. Satava RM. Medical applications of virtual reality.J Med Syst. 1995;19:275–280.
5. Burdea G, Deshpande S, Popescu V, et al. Comput-erized hand diagnostic/rehabilitation system using a force feedback glove.Stud Health Technol Inform. 1997;39:141–50.
6. Emmelkamp PM, Bruynzeel M, Drost L, van der Mast CA. Virtual reality treatment in acrophobia: a com-parison with exposure in vivo.Cyberpsychol Behav. 2001;4(3):335–339.
7. Rothbaum BO, Hodges LF, Anderson PL, Price L, Smith S. Twelve month follow-up of virtual reality and standard exposure therapies for the fear of fly-ing.J Consult Clin Psychol. 2002;70:428–432. 8. Rothbaum BO, Hodges LF, Alarcon R, et al. Virtual
re-ality exposure therapy for PTSD Vietnam veterans: a case study.J Traumatic Stress. 1999;12:263–272. 9. Riva G. Virtual reality and body experience: a new
approach to the treatment of eating disorders.Int J Virtual Reality. 1999;2(2):9–16.
10. Hoffman HG, Patterson DR, Carrougher GJ, Sharar SR. Effectiveness of virtual reality based pain con-trol in multiple treatments.Clin J Pain. 2001;17:229– 235.
11. Rizzo AA, Buckwalter JG, Neumann U. Virtual real-ity and cognitive rehabilitation: a brief review of the future.J Head Trauma Rehabil. 1997;12:1–15. 12. Schultheis MT, Rizzo AA. The application of virtual
re-ality technology for rehabilitation.Rehabil Psychol. 2001;46:1–16.
13. Wilson BA. Ecological validity of neuropsychological assessment: do neuropsychological indexes predict performance in everyday activities?Appl Prev Psy-chol. 1993;2:209–215.
14. Luria A.The Working Brain: An Introduction to Neuropsychology(trans. B Haigh). New York: Basic Books; 1973.
15. Crawford J. Assessment of attention and executive functions.Neuropsychol Rehabil. 1998;8:209–211. 16. Elkind J. Uses of virtual reality to diagnose and
ha-bilitate people with neurological dysfunctions. Cy-berPsychol Behavior. 1998;1:263–274.
17. Pugnetti L, Mendozzi L, Attree E, et al. Probing memory and executive functions with virtual
real-ity: Past and present studies.CyberPsychol Behavior. 1998;1:151–162.
18. Pugnetti L, Mendozzi L, Motta A, Cattaneo A, Barbieri E, Brancotti S. Evaluation and retraining of adults’ cognitive impairments: which role for virtual reality technology?Comput Biol Med. 1995;25:213– 227.
19. Pugnetti L, Mendozzi L, Brooks BM, et al. Ac-tive versus passive exploration of virtual environ-ments modulates spatial memory in MS patients: a yoked control study.Ital J Neurol Sci. 1998;19:S424– S430.
20. Mendozzi L, Motta A, Barbieri E, Alpini D, Pugnetti L. The application of virtual reality to document coping deficits after a stroke: report of a case.CyberPsychol Behavior. 1998;1:79–91.
21. Elkind JS, Rubin E, Rosenthal S, Skoff B, Prather P. A simulated reality scenario compared with the computerized Wisconsin Card Sorting test: an anal-ysis of preliminary results.CyberPsychol Behavior. 2001;4:489–496.
22. Pugnetti L, Mendozzi L, Attree EA, et al. Probing memory and executive functions with virtual reality. CyberPsychol Behavior. 1998;1:151–161.
23. McGeorge P, Phillips LH, Crawford JR, et al. Using vir-tual environments in the assessment of executive dys-function.Presence: Teleoperators Virtual Environ. 2001;10:375–383.
24. Sohlberg MM, Mateer CA.Introduction to Cognitive Rehabilitation. New York: Guilford Press; 1989. 25. Wann JP, Rushton SK, Smyth M, Jones D. Virtual
envi-ronments for the rehabilitation of disorders of atten-tion and movement. In: Riva G, ed.Virtual Reality in Neuro-Psycho-Physiology. Amsterdam: IOS Press; 1997:157–164.
26. Rizzo AA, Buckwalter JG, Humphrey L, et al. The virtual classroom: a virtual environment for the as-sessment and rehabilitation of attention deficits. Cy-berPsychol Behavior. 2000;3(2):483–501.
27. Rizzo A, Buckwalter J, Neumann U, et al. Virtual environments for targeting cognitive processes: an overview of projects at the University of Southern California Integrated Media Systems Center. Pres-ence: Teleoperators Virtual Environ.2001;10:384– 400.
28. Kennedy RS, Lane NE, Berbaum KS, Lilienthal MG. Simulator sickness questionnaire: an enhanced method for quantifying simulator sickness. Int J Aviat Psychol. 1993;3:203–220.
29. Lengenfelder J, Schultheis MT, Al-Shihabi T, DeLuca J, Mourant R. Divided attention and driving: a pi-lot study using virtual reality technology. J Head Trauma Rehabil. 2002;17:26–37.
30. Carpenter JP, Just M. Spatial ability: an information processing approach to psychometrics. In: Sternberg RJ, ed.Advances in the Psychology of Human Intel-ligence, ed. 3. Hillsdale, NJ: Erlbaum; 1986. 31. Astur RS, Oriz ML, Sutherland RJ. The
characteriza-tion of performance by men and women in a virtual morris water task: a large and reliable sex difference. Behav Brain Res. 1998;93:185–190.
32. Jacobs WJ, Laurance HE, Thomas K. Place learning in virtual space I: Acquisition, overshadowing and transfer.Learning Motivation. 1997;28:521–541. 33. Jacobs WJ, Laurance HE, Thomas K. Place learning
in virtual space II: topographical relations as one measure of stimulus control.Learning Motivation. 1998;29:288–308.
34. Sandstrom NJ, Kaufman J, Heuttel SA. Males and fe-males use different distal cues in a virtual environ-ment navigation task. Cognitive Brain Res. 1995; 6:351–360.
35. Morris R. Spatial localization does not require the presence of local cues. Learning and motivation.Int J Neurosci. 1981;48:29–69.
36. Skelton RW, Bukach CM, Laurance HE, Thomas KG, Jacobs J. Humans with traumatic brain injuries show place learning deficits in computer generated vir-tual space.J Clin Exp Neuropscyhol. 2000;22:157– 175.
37. Stanton D, Foreman N, Wilson P. Uses of virtual re-ality in clinical training: developing the spatial skills of children with mobility impairments. In: Riva G, et al, eds. Virtual Environments in Clinical Psy-chology and Neuroscience. Amsterdam: IOS Press; 1998:219–232.
38. McComas J, Pivik J, Laflamme M. Children’s transfer of spatial learning from virtual reality to real environ-ments.Cyberpsychol Behav. 1998;1:121–128. 39. Cromby J, Standen P, Newman J, Tasker H.
Success-ful transfer to the real world of skills practiced in a virtual environment by student with severe learning disabilities. In: Sharkey PM, ed.Proceedings of the 1st European Conference on Disability, Virtual Re-ality and Associated technologies. Maidenhead, UK; 1996:305–313.
40. Rizzo AA, Schultheis MT, Rothbaum B. Ethical is-sues for the use of virtual reality in the psychologi-cal sciences. In: Bush S, Drexler M, eds.Ethical Is-sues in Clinical Neuropsychology. Lisse, NL: Swets & Zeitlinger Publishers; 2002.
41. Shepard RN, Metzler J. Mental rotation of three-dimensional objects.Science. 1971;171:701–703. 42. Vandenbergm SG, Kluse AR. Mental rotations, a
group test of three-dimensional spatial visualization. Percept Motor Skills. 1978;47:599–604.
43. Atkinson RC, Shiffrin RM. Human memory: a pro-posed system and its control processes. In: Spence KW, ed.The Psychology of Learning and Motiva-tion: Advances in Research and Theory. Vol 2. New York: Academic Press; 1968.
44. Baddeley A.Working Memory. Oxford: Clarendon Press; 1986.
45. Rose FD, Attree EA, Johnson DA. Virtual reality: an assistive technology in neurological rehabilitation. Curr Opin Neurol. 1996;9:461–467.
46. Attree EA, Brooks BM, Rose FD, Andrews TK, Lead-better AG, Clifford BR. Memory processes and virtual environments: I can’t remember what was there, but I can remember how I got there. Implications for people with disabilities. In: Sharkey P, ed. Proceed-ings of the First European Conference on Disability, Virtual Reality, and Associated Technology. Maiden-head, UK: University of Reading; 117–121. 47. Peruch P, Verhcer J, Gauthier GM. Acquisition of
spa-tial knowledge through visual exploration of simu-lated environments.Ecol Psychol. 1995;7:1–20. 48. Brooks BM, McNeil JE, Rose FD, Greenwood RJ,
At-tree EA, Leadbetter AG. Route learning in a case of amnesia: a preliminary investigation into the efficacy of training in a virtual environment.Neuropsychol Rehabil. 1999;9:63–76.
49. Brooks BM, Attree E, Rose FD, Clifford B, Leadbet-ter AG. The specificity of memory enhancement dur-ing interaction with a virtual environment.Memory. 1999;7:65–78.
50. Strickland D, Marcus LM, Mesibov GB, Hogan K. Brief report: two case studies using virtual reality as a learning tool for autistic children.J Autism Dev Dis-ord. 1996;26:651–659.
51. Strickland D. Virtual reality for the treatment of autism.Stud Health Technol Inform. 1997;44:81– 86.
52. Inman D, Peaks J, Loge K, Chen V. Teaching or-thopedically impaired children to drive motorized wheelchairs in virtual reality.Proceedings from the 2nd Annual International Conference on Virtual Reality and Persons with Disabilities. San Francisco, CA; 1993.
53. Christiansen C, Abreu B, Ottenbacher K, Huffman K, Masel B, Culpepper R. Task performance in vir-tual environments used for cognitive rehabilitation after traumatic brain injury.Arch Phys Med Rehabil. 1998;79:888–892.
54. Schultheis MT, Rizzo AA. The virtual office: assessing & re-training vocationally relevant cognitive skills. Proceedings from the 9th Annual Medicine Meets Virtual Reality Conference, Newport Beach, CA; 2002.
55. Liu L, Miyazaki M, Watson B. Norms and validity of the DriVR: a virtual reality driving assessment for persons with head injuries. Cyberpsychol Behav. 1999;2:53–67.
56. Schultheis MT, Mourant RR. Virtual reality and driv-ing: the road to better assessment of cognitively im-paired populations.Presence: Teleoperators Virtual Environments. 2001;10:436–444.
57. Brown DJ, Kerr SJ, Bayon V. The development of the Virtual City: a user centered approach. In: Sharkey P, Rose D, Lindstrom J, eds.Proceedings of the 2nd European Conference on Disability, Virtual Reality and Associated Techniques. Skove, Sweden Univer-sity of Reading; 1998: pp. 11–16.
58. Cobb SVG, Neale HR, Reynolds H. Evaluation of vir-tual learning environments. In: Sharkey P, Rose D, Lindstrom J, eds.Proceedings of the 2ndEuropean Conference on Disability, Virtual Reality and As-sociated Techniques. Sk¨ove, Sweden: University of Reading; 1998:17–23.
59. Mowafty L, Pollack J. Train to travel.Ability. 1995;15: 18–20.
60. Jones LE. Does virtual reality have a place in the reha-bilitation world?Disabil Rehabil. 1998;20:102–103.
61. Korpela R. Virtual reality: opening the way.Disabil Rehabil. 1998;20:106–107.
62. Wilson PN, Foreman N, Stanton D. Virtual reality, disability and rehabilitation.Disabil Rehabil. 1997; 19:213–220.
63. Foreman N, Orencas C, Nichols E, Morton P, Gell M. Spatial awarenes in 7- to 11-year old physically handi-capped children in mainstream schools.Eur J Special Needs Educ. 1989;4:171–178.
64. Whalley LJ. 1996 Ethical consideration in the applica-tion of virtual reality to medicine.Comput Biol Med. 1989;25:107–114
65. Bloom RW. Psychiatric therapeutic applications of virtual reality technology (VRT): research prospectus and phenomenological critique.Stud Health Technol Inform. 1997;39:11–16.
66. Ring H. Is neurological rehabilitation ready for ’im-mersion’ in the world of virtual reality?Disabil Re-habil. 1998;20:98–101.
67. Rothbaum BO, Hodges LF, Kooper R, Opdyke D, Williford JS, North M. Effectiveness of computer-generated (virtual reality) graded exposure in the treatment of acrophobia. Am J Psychiatry. 1995; 152:626–628.
