Initial work involved developing simple experiments and prototypes to explore the ca- pabilities of the ARToolKit. The experience gained in building these was invaluable in later work on more complex interaction metaphors.
used to trigger interaction events in AR systems. Different types of behaviours and gestures were considered, including marker cards’ position in relation to each other, distances between different markers, marker occlusion and orientation. When possible, the prototypes that were developed with the ARToolKit focused on AR information display techniques for museums. This section describes the different test systems that were implemented.
4.2.1 Position
(a) Stack of books loaded in ARToolKit
(b) Select previous book (c) Select next book
Figure 4.4: Simple position interaction
The first prototype used the results of the SOFAR (Moreau et al., 2000) Dynamic CV Agent developed during the third “Agent Fest” held in the IAM group in November 2000. The Dynamic CV agent collects all sorts of information about people from other agents in the SOFAR agent framework. The agent was flexible enough for its output to be used in various ways, such as rendering to the web or into an AR application. The type of query used in the prototype returned the publication list for someone in the group. These results were used to generate a VRML model of a pile of books, with each book representing a publication. As the publication title may not be readable on the book’s spine, it was decided that a floating text box would display the selected publication’s full details. The model would be loaded into the ARToolKit where the user would view it, as shown in Figure 4.4(a).
This raised a problem on how to select each publication. One solution was to track a marker card to cycle through the stack of books. By placing the marker card on one side of the stack card, the next book would be selected and its details highlighted on the floating text box. Putting the marker card on the other side would select the previous book. This is shown in Figure 4.4(b) and Figure 4.4(c).
4.2.2 Distances Between Markers
A second prototype was developed that looked at the distances between markers relative to their orientation; for example markers may need to be the right way up to trigger an event in some applications.
(a) AR jigsaw (b) Connecting two pieces (c) All pieces connected
Figure 4.5: Position interaction depending on the marker’s orientation
The prototype developed is essentially a jigsaw puzzle in AR, as can be seen in Figure 4.5. There are three marker cards; each card holds one of the three virtual jigsaw pieces. Placing two cards next to each other in the correct order and the right way up will join the two pieces. When all three are together in the correct order the three pieces join up. This prototype showed that distances between markers can be used to control the inter- actions between them.
4.2.3 Occlusion
(a) Model (b) When the button is pressed, labels are placed on the model.
Figure 4.6: Buttons triggered by marker occlusion
When a marker is occluded or not visible, the ARToolKit will not be able to draw it. Monitoring when certain markers vanish and appear could be used to trigger events and mimic the action of buttons. For example, when the user occludes the button in
Figure 4.6 labels appear around the object. This simple technique could potentially perform all manner of user interactions.
However, the ARToolKit is very sensitive to occlusion: the whole marker must remain in view at all times to be identified by the tracking system. Users accidentally obscuring a corner or part of the marker will cause tracking errors, such as flickering, the wrong object being displayed or even complete loss of tracking. This can be frustrating as the marker may seem visible to the user but due to one obscured corner no object is displayed. Various other conditions may also cause occlusions, such as shadows, swift movement or poor lighting conditions. Any of these actions could trigger the button, producing interface events that the user was not expecting. This sensitivity to error is obviously not acceptable for user interfaces.
Current research in tracking techniques may result in more occlusion tolerant marker- based AR systems. There is even the possibility that these techniques will be able to track occluding objects, for instance hands or fingers, as they move over markers. If this is possible users will be able to use their fingers to interact with user interface elements, such as buttons and sliders, that are overlaid on the marker (Malik et al., 2002; Gordan et al., 2002).
Another possibility is to integrate different types of tracking technologies, such as vision and magnetic trackers. This has been explored in certain projects, such as the Personal Interaction Panel described in Chapter 2. Unfortunately this would increase the cost and complexity of the AR system.
4.2.4 Orientation
The orientation of marker cards was also used to trigger user interface events. Figure 4.7 shows a text box projected onto a card. When the user presses the button and tilts the card down, the text scrolls down; when the card is tilted up the text scrolls up. Similar techniques have also been explored for scrolling on handheld displays (Rekimoto, 1996a).
(a) Text box (b) Scrolling down (c) Scrolling up
This idea was also used for panning around an image, as shown in Figure 4.8. The image can be panned both vertically and horizontally.
(a) Image panning (b) Feature of an image
Figure 4.8: Panning around an image
This approach removes the need to click on scroll bars or buttons to pan around text or images. This is an effective use of markers’ physical properties as a user interface control, and is more suited to the tangible AR environment than, for example, tra- ditional scrollbars activated by clicking on buttons. There are possible advantages in mobile applications; for instance, a wearable computer user may have trouble accurately manipulating a traditional scrollbar on a HMD or hand held PC if they are walking. Although this technique was not used directly in later work, it could still be useful in certain applications involving text or images projected onto markers. The largest problem is determining when to enable scrolling: with the prototype, users obscured the button to start scrolling. As explained above, this approach is too prone to error to be effective in a user interface. Another problem is that to perform scrolling with orientation, a frame of reference is needed. As the ARToolKit only calculates the position of markers in relation to the camera, it is impossible to determine what way the user wishes to scroll from the marker’s orientation alone. Solutions to this problem may involve background markers, for instance on a table or on the floor, to act as a frame of reference. If when a user wants to start scrolling could be determined, differences in orientation from that time could be used for scrolling. A very simple solution would be to scroll only in relation to the camera, so markers parallel to the camera plane would be still and scrolling is affected by the deviation in orientation from that point.