Augmented Reality Tracking

Tracking: 3D-environment

The most challenging aspect of augmented reality is tracking. Because the prototype obtains camera footage to which it should align virtual objects, a reliable way to achieve an augmented reality experience is with tracking visually. This means that software is processing camera footage, and maps the 3D environment that is filmed by the device. Since this project has its focus on how to use an augmented reality tool rather than the focus on how to track a 3D environment for augmented reality purposes, it has been decided to use external software for the augmented reality tracking. A company that provides a reliable library for Unity3D is Vuforia. From their website packages can be downloaded, which can be implemented within the Unity3D project. Additionally, Vuforia includes a wide variety of documentation which could make a development process quicker.

The Vuforia library is able to recognize five kinds of real world objects:

1. 2D-images, uploaded to the Vuforia-engine. Uploading can also be done via an IPA-call, meaning this could be done automatically.

2. Small 3D objects, scanned with the Vuforia Object Scanner. This is an application for the Samsung Galaxy S5 and the Google Nexus 5 [21]. After scanning the object, the scan should be uploaded to the Vuforia website.

3. Cilinders. 4. Cubes.

5. VuMarks, which are 2D-images that can be displayed in the real world with a small size and can include digital information like a string or a number of bits.

Since the product should fit as easy as possible into the daily work of the technician. This means that adding additional scan objects to the technicians’ environment would include additional practices for ordering and placing scan objects to machine areas. In theory it would be ideal when the software could recognize and track 3D perspectives from the machines. However, with the Vuforia library it is only possible to scan small real world objects. Additionally, the scanning of 3D objects is not the most easy practice and therefore less reliable to include in the technicians daily practices. A solutions that includes both aspects is that the software is tracking the warning stickers that are already included in most machine areas. These warning could for example be warnings for electrical shocks or messages to wear earplugs. Warnings that were used during prototyping with this way of tracking are displayed in figure 6.19. Because the warning given in figure 6.19.a did not include enough recognizable points for the Vuforia software, experimenting is executed with recognizable patterns behind the warning, like shown in figure 6.19.b. This appeared to be reliable enough to track on 3D perspectives.

Fig. 6.19 - Warning messages that are (a) not reliable for tracking and (b) reliable for tracking with Vuforia

Tracking: machine number

Although the warning messages are recognizable with the Vuforia software, all stickers and thus all machine areas look similar for the software. This resulted in virtual objects that were drawn at a certain machine and remained visible when another machine was scanned afterwards. A way to include information within a 2D-trackable object in the Vuforia library are VuMarks. These objects work like QR-codes, but include more visual information. An Adobe Illustrator project that is provided by Vuforia let developers create their own VuMarks. Within these VuMarks a required amount of elements should be included that can be colored white or black. The combination of certain elements being white or blue indicates the software with what string or number it corresponds. Because strings require a large number of elements, and only machine numbers are necessary to be read from the VuMarks, it is decided to include only numbers within the element combinations. Within figure 6.20.a it can be seen how the VuMark looks that is used for this project. At the positions of the pink circles the Vuforia database can generate white or black blocks. A combination of these 28 blocks indicates a bit between one and ten to the tracking software. For example, figure 6.20.b corresponds to the number one, while figure 6.20c corresponds to the number two. The C# code that reads the number from a recognized VuMark is given in appendix C.2.

Fig. 6.20 - The VuMark that is designed for this project

Object builder

When the user of the application touches the screen, and is not pressing a user interface element, the software draws a virtual element to the 3D environment. Depending on what the user has selected as drawing object and color, the software draws a “prefab” to the position that the user presses. When a circle is selected as drawing object, the software remains drawing circles that will form lines eventually. When a message is selected as drawing object, the message prefab is only instantiated when the user clicks the screen. This means that messages will not remain being created while the user holds its finger on the screen. With the Unity3D standard function “ScreenToWorldPoint”, the software can calculate the finger touch to a position in the 3D environment. Because it is expected that all objects should align with the machine, the virtual objects are instantiated at a distance that the VuMark is from the camera.

Whenever a new object is created, the software creates the prefab as a child of the VuMark, meaning that it will move together with the tracked VuMark in the camera view. This causes that all virtual objects will remain aligned with the machine, and will create an augmented reality experience. In order to only display virtual objects at one machine, instead of at all machines, the virtual objects are associated with only one machine. When an object is created, its name is changed to start with the machine number. For example, if the user creates a blue circle at machine one, the software will change the circle’s name to “1-BlueCircle”. At all times, the software is using a script that only displays the virtual objects which names start with the same number as the recognized machine area. Additionally, when the user presses the delete button, only the virtual objects that start with the same number as the recognized machine area are being deleted. C# scripts of these functionalities can be found in appendix C.2.

Object builder from network

When the application is connected to the assisting application, it can receive virtual objects from the other device. These objects are handled the same way as objects that are created by the user, as described in the previous paragraph. However, the coordinates of the received objects are being adapted to fit the screen size first, before these objects are instantiated. How this is done is explained at the networking paragraph.

An example of how virtual objects are displayed on in the application is given in figure 6.21. This image includes objects that are created by the user as well as by the assisting technician. Additionally, it can be seen that the software uses the corresponding author name, together with the actual date and time, so that the user could identify the messages. C# scripts that are designed for these functions are given in appendix C.2.

Fig. 6.21- Screen of the application including virtual objects from the user and the assisting technician