Augmented Reality for Remote Access Laboratories

Augmented Reality varies as a consequence of the framework it operates within. Augmented Reality for gaming and entertainment has very different capabilities and needs. Specifically, for Remote Access Laboratories, Augmented Reality exists as a technology to support student interaction with the didactic experiment. This may manifest itself as a series of visual overlays on the real-time video streams supplied from camera’s monitoring the experiment unfolding. Object models for both AR and RAL are discussed and catalogued to understand the requirements of any future supporting framework. This research work focuses on the Vision Analysis module of the Augmented Reality sub-system.

The amalgamation of Augmented Reality and Remote Access Laboratories aims to provide students with an environment that enhances their experience of accessing laboratory resources from a distance. Current properties of an AR environment, within the RAL framework, mostly consist of visual sensory feedback in the form of computer enhanced live video streams. Data delivered from remote experiments, including the live video stream, are used to create computer generated imagery which is overlayed with the video stream to provide the student with additional information. When the computer-generated imagery and live video stream are properly coordinated, the student can become involved at a deeper level with the learning exercise. The immersive environment is conducive to improved pedagogical outcomes [50].

Engaging students through new learning tools and methods such as quest-based gamification [51] has provided a rich learning environment which has become accepted as the new normal. Interactive sensory feedback may manifest itself in many forms and from numerous sources within the experimental rig. Figure 1-3 [52] demonstrates a simple but effective remote laboratory application of a microprocessor centric control system. The left image (A) appears as the user views it, with virtual overlaid objects included to improve the student’s interaction with the experiment. The right image (B) consist of a large fiducial marker to identify and orientate the AR system. This system provides the user with computer generated objects which are combined with reality in such a way that the user perceives them as part of the environment. Interaction with the virtual objects initiates actions that would be expected from an in-situ action. The use of virtual visual images within the video stream is an effective AR application to a control systems remote experiment.

Aside from improving student engagement with the content, AR provides other important aspects. Many current RAL systems convey only enough data to validate theoretical models, but do little to familiarise the students with the equipment they may be using once they graduate. For example, nursing students struggle to gain physical access to important equipment they are expected to be familiar with once on-site in hospitals. Remote laboratory systems have helped to provide nursing students with online access to the equipment, which allows them to establish and maintain familiarity and confidence with the equipment. However simply clicking an on-screen virtual button, representing a function, may not be sufficient for the student to become

(A) User View (B) Actual View

competent with the device [53]. It is not always about the function, but how the equipment responds. While a message on a display might appear, operationally, other stimuli may also be present, such as sounds to indicate the current operating state. Fully engaging the student’s senses creates an invaluable experience that supports not just the students’ confidence in the theoretical validation, but the familiarization with their tools of trade. Associating computer generated sensory information to user interactions with the remote experiment becomes a critical AR function; this is critical because our understandings of theory and device operations exist in many different contexts. An augmented reality system must interpret real-world data from the environment, and as such must then acquire some understanding the various input signals. An Analytical Control System (ACS), depicted in Figure 1-4, perceives the real-world and derives limited understanding. The ACS receives raw data from the remote experimental rig, including sensor data and the live video stream, to develop information sets regarding the state or processes being performed. Inputs may consist of analogue and digital signals from the rig, plus secondary data from other sources such as GPS sensors. Audio has been shown as an important source of data, reflecting the sounds and noises expected from the operation of equipment. Other sensory information, such as tactile data that could be expected from sensing temperature combines to improve the sense

of being present with the rig. Both a priori and/or posteriori knowledge shapes the conclusions of the ACS, and creates computer generated feedback to the user.

An experimental rig provides a range of data signals to the user, and the goal of the data is confirmation of the theoretical lessons. But the data can also be used to improve the method in which the didactic proof is delivered. Shown in Figure 1-5, an AR system may accept information from a number of physical laboratory resources. The ACS real world feedback in Figure 1-4 becomes the Virtual Object Generator (VOG) and determines the necessary virtual objects to produce as well as managing synchronisation between real and virtual world objects. Enhanced outputs consist of virtual objects that become the source of sensory enhancement to immerse the student into the laboratory environment.

The simplified AR RAL interface model of Figure 1-5, accounts for all categories of data streams. Input data arrives from RL devices such as thermocouples, strain gauges, etc and provides key information to the VOG. Mechanical or tactile/haptic inputs to the model arrive from human interface devices such as sensor load gloves [54] and are also inputs to the VOG. Both two or three-dimensional video streams require sub-processing by the Vision Analysis (VA) system. Depth data is vision data retrieved from 3D vision source such as gesture sensing components [55]. Any data extracted from the VA system is applied to the VOG. The VA system is the focus of these works. Computer

generated information, pertaining to the experiment, is built and synchronised by the VOG and dispatched back to the user, incorporating both real and virtual objects. Different senses are engaged for each of the three outputs of Figure 1-5. The sense of touch for effects such as tension, force [56] or temperature [57] are commandeered, and the user may feel the forces necessary to turn a handle, or the heat from a chemical process. Spatially synchronised computer-generated audio signals are powerful to give the student a three-dimensional sense of the environment. Of the three output augmentations, vision is arguably the most important, as it is the primary source of our understanding of the real world.

This interactive feedback may manifest as synthetic input devices or measurements, such as instances of virtual stimulus or measurement devices. Purpose built hardware devices are also developed to act as either source or sink of the interactive systems.

1.4.1 Student Engagement

Enhancing video streams through the inclusion of virtual objects is intended to improve the user’s engagement with the experiment, yet there is a concern that technology may get in the way of learning outcomes [58, 59]. Research has demonstrated that AR enhanced RAL creates an environment in which the students become immersed with the experiment [60]. It has been speculated that AR RAL systems provide a scenario more closely matching the hands-on experience. Improvements in technology allow high quality sensory feedback to the video stream. The psychological effect of quality immersion into the environment is then linked to improved learning outcomes [61]. Additionally, there is research which indicates that employing AR sub-systems affects students motivation and satisfaction [62]. The use of augmented systems must improve the student’s experience, and not become a burden to the pedagogical outcomes.

1.4.2 Concept of Presence

While technology supports and interfaces the remote systems to the remote user, a sense of presence in-situ is how student engagement evolves. Ideally, augmented reality will mitigate perception of the technology interface. That is, in the RAL situation, the user will not sense the use of technology to perform the experiments. From works of IJsselsteijn et al. [63], key considerations define the concept of presence.

• Prominent and valuable sensory feedback is presented to the user in an appropriate manner,

• Feedback between users’ actions and the activation of remote devices must respond in real-time,

• Data presented to the users must be consistent with the nature of the objects and respond as expected.

Ignoring the concept of presence when developing AR RAL systems risks the loss of all AR enhancements. Tracking failures, timing and synchronisation errors, plus the poor application of virtual objects (as listed above) destroy the continuity with reality.