Hardware Platform

Ideally, our intended hardware platform is a smartphone and Google Glass i.e. an independent head mounted computer. Since Google Glass is currently unavailable, a considerable amount of time had been spent in assembling a prototype system similar to the intended target platform.

The first attempt involved connecting a HMD to a smartphone where the HMD would display the information shown on the smartphone’s screen. This smartphone was connected via Bluetooth to another phone that was held in the users hand and could be used for providing gestural input. The difficulty in connecting external devices (such as cameras and orientation sensors) to a smartphone prohibited us from using this setup. However, this preliminary setup helped understanding that Bluetooth is not fast enough for transferring touch positions between two mobile devices.

The above-mentioned difficulties led us to look at Ultra Mobile PC’s (or UMPCs) as a possible alternative. UMPCs can connect to external devices (like HMDS and cameras) through USB and smartphones through wireless networks. These devices are lightweight, and can be easily worn using small bags (see Figure 4.1). Additionally, UMPCs run the Windows operating system that makes it easier for programs written on desktop PCs to be ported to the UMPC.

Figure 4.1 : A UMPC and user wearing the system !

4.1.2 HMD selection

Our HMD selection criteria required the HMD to be monocular and to preferably have a head orientation tracker. The monocular HMD is preferable because it allows people to more easily see the real world, while still viewing additional information on the HMD. This type of HMD is also useful for mobility. The Brother Air Scouter HMD 2,(see left of Figure 4.2) fulfills one of the above criteria and uses retinal projection display technology similar to Google Glass, which is why it was considered for usage with our system. However, the absence of an integrated Tracker made it difficult to use this HMD with our !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! 2_{!http://www.brother.com/en/news/2011/airscouter/!} & ! HMD& & & & & & & & HHD& & & & & Webcam& & & & & & & & UMPC&

system. The Vuzix iWear 900 3(see right of Fig. 4.2) has an integrated head tracker and can be easily modified from binocular to a monocular display. Although the display technology was not as good as the Air Scouter, the modified Monocular Vuzix HMD was used for the prototype system since it fulfilled our criteria.

!

4.1.3 Software platform

The initial prototype system used two Android smartphones that communicated with each other through Bluetooth. One phone had a combination of a Map View that showed points of interest on a map and a List view that listed information about those points of interest. The other phone, that we planned to feed video signal to a HMD, had a dedicated AR view that tracked the position of the phone using GPS and augmented virtual models on the !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!

3!http://www.vuzix.com/consumer/products_vr920.html!

Figure 4.2: The Brother AirScouter and Vuzix HMD

Camera Feed corresponding to the points of interest. The HITLabNZ Outdoor AR library4 was used to prototype this system.

Map View AR View

Figure 4.4 : Initial platform with Map and ARView on different HHDs

Moving to a smartphone-UMPC system required a complete change in software platform. Choosing a portable software platform was necessary so that we could easily migrate our code. Prototyping different applications quickly without sacrificing robustness was also an important criteria, which prompted us to look at creative coding platforms. Creative coding platforms allow the developer to quickly prototype programs, without any knowledge of low-level libraries. This simplifies programming and facilitates rapid prototyping. For the prototype system, the author quickly prototyped applications without an in-depth knowledge of OpenGL, Model Loading libraries, Computer Vision etc.

Initially, the Processing platform [34] was chosen because of its ease of use and the speed with which one could prototype ideas. Because Processing does not offer real-time performance on a UMPC, the Openframeworks platform was finally chosen. Openframeworks [35] is a creative coding platform that uses C++ and is much faster than Processing which is Java based. Its real time performance on the UMPC and portability to multiple platforms like iOS, Android, Windows, Linux and Mac OS X were suitable for the prototype system.

!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!

4.1.4 Communication

The most important feature required in our HMD-HHD hybrid system was that the positions of the user’s fingers must be transmitted in real time and mapped correctly to the users HMD – which is why choosing the right communication protocol was critical for our system. Both speed and accuracy were desired since any delay in transmitting the touch positions would affect the user experience and could lead to incorrect selection of virtual content shown on the HMD.

The User Datagram Protocol (UDP) was chosen because of its speed. By shortening the length of the UDP messages, transmission of finger positions in real time was possible without any problem of lost packets/messages. Since smartphones are capable of hosting mobile hotspots and sending and receiving messages over the hosted network without any attenuation, no external network was required.

4.2 Software Architecture

The software of the prototype system consists of two modules: one on the smartphone and the other on the PC (see Figure 4.5). The software module on the smartphone captures user interaction and sends this information to the module on the PC. The software module on the PC draws the virtual scene while the user interaction information from the smartphone is processed.

!

Figure 4.5 : Software architecture of the prototype system !

! !

Touch Event Detector

Touch Classifier UDPMessageCreation ModelLoader& XMLParser& AssetsLoader! FilterTouchInfo& Rendering!Module! HHDtoScreenCoords& Action&Module& UDPMessageParser& TouchModule!

HHD

HMD

4.2.1 HHD Modules

The Touch Event Detector Module detects finger touches on the smartphone that are sent to the Touch Classifier. The Touch Classifier determines the type of Gesture performed and creates a Message. This message contains details like number of Fingers, type of Gesture Performed, specific details of the Gesture (like duration of the touch while gesture was performed) etc. The touch positions are then packaged into a message, which is sent to the PC by the UDPMessageCreation Module.

!

4.2.2 PC Modules

The state of the different models present in the virtual scene are stored in an XML file that is parsed using an XMLParser and used by the ModelLoader to load the models and set its properties like scale, position and rotation.

The PC receives the message sent by the smartphone through the UDPModule and the message is parsed and filtered by the FilterTouchInfo module to get information regarding the touch positions. This information is used by the HHDtoScreenCoords module to convert the touch positions from HHD screen coordinates to HMD screen coordinates. The touches are then sent to the Rendering Module that draws them as circles on the user’s HMD view.

Information about any gesture performed on the HHD is filtered from the UDPMessageParser by the Action Module and is used with the position of virtual objects provided by the Rendering Module to create an Action Event. This action event could be selection, translation or scaling the virtual object depending upon the nature of the target application and is sent to the Rendering Module.

4.3 Interaction Design