System re-configurations

Later stages of the experimental work of the thesis required that modifications be made to the basic system set-up to allow for the experimental comparison of different system configurations. When constructing remote gesture technologies there are various factors which can be modified to affect how the device works. Two key factors of remote gesture system design were identified as target areas for experimental modification.

The first factor was the location of the gesture output (to the person located with the task artefacts), with the remote gestures being either projected directly into a task space or

alternatively presented to the remote worker on an external VDU showing an overlay of the workspace and the gestural information (these represent the two current trends in remote gesture tool design). A second factor that can be modified is the format of the gesture. For the experiments in this thesis three different formats of gesture, unmediated representation of hands, unmediated hands with a sketch facility and digital sketch only were explored. Mentioned further in section 5.3.1 the basic premise for the inclusion of these formats being that this thesis has chosen on the basis of prior research evidence to focus more closely on rich forms of gesture representation, which have been shown to be of particular benefit in collaborative physical tasks (consequently excluding from consideration simplified gesturing systems such as the laser dot systems sometimes explored). The combination of these location and format factors gives six different possible system configurations, displayed in table 3.1 (these six combinations represent the specific conditions that were compared in subsequent experiments).

Gesture Format

Gesture Location

Projected TV

Hands only Projected hands TV hands Hands and sketch Projected hands & sketch TV hands & sketch Digital sketch only Projected sketch only TV sketch only

Table 3.1 Comparison of possible gesture locations and formats

To construct these various system configurations low-tech prototypes were assembled, modified from the basic set-up discussed above (as shown in figure 3.1).

The modification that was incorporated to allow the workers to see gestures on a separate video window, as opposed to the previously used projection, was the removal of the video projector, and the inclusion of a second TV (one for the Worker as well as the Helper). To create the effect of remote gesturing, a 4-channel video mixer unit was incorporated (Videonics MX-1) and using the manual T-bar control, a 50% transition image was created of both video feeds overlapped. This mixed live video feeds from above both the Worker and Helper desks (see figure 3.6a for schematic and 3.6b for an annotated image of a typical TV overlay view). This enabled the Helper‟s to guide their hand movements and gestures relative to the shared task artefacts by looking at a video window, exactly as in the original system set- up.

Figure 3.6a Video presented Hands (schematic)

Figure 3.6b Video presented Hands (screen capture)

The system was extended for analysis of the „Hands & Sketches‟ condition, by facilitating remote sketching by giving the Helpers a board marker pen, and allowing them to write on the surface over which they were gesturing (which was a dry-wipe whiteboard). The resultant sketches either A) being projected onto the workers desk (see figure 3.7) or B) video mixed over their video feed of the task space (see figure 3.8). Example illustrations of the gesture output for each of these systems can be seen in figure 3.9.

Video Camera Video Camera Worker Helper TV TV Mixer

Remote Helper Hands Local Worker Hands

Figure 3.7 Projected Hands & Sketches

Figure 3.8 Video presented Hands & Sketches

Figure 3.9 Projected (left) and Video (right) presented Hands & Sketches (in each case image captured from Helper‟s TV view)

Video Camera Video Camera Projector Worker Helper TV Whiteboard Video Camera Video Camera Projector Worker Helper TV TV Mixer Whiteboard

To produce a „Sketches only‟ configuration a significant change was required in the technology. As a video feed from above the Helper‟s desk was no longer required the camera that normally occupied this position was removed. The gesturing / sketching surface that had previously been used was also replaced with an A2 sized Wacom Tablet. The tablet was connected to an IBM Pentium III PC (with 547 MHz CPU, 256MB RAM, and a 32MB Matrox Millenium G400 graphics card) running MS Paint V.5.1 on Windows XP. The PC was set to an output resolution of 1024x768 pixels in 32bit colour. The output of this paint program was then presented to the Worker through a projection onto their desk (see Figures 3.10a+b) or presented mixed over a live video feed of their task space (see Figure 3.11a+b).

Figure 3.10a Projected Sketches only (schematic)

Figure 3.10b Projected Sketches only (screen capture)

Video Camera Projector Worker Helper TV Whiteboard PC Wacom tablet Worker‟s Hands Assembly pieces

Last Sketch Cursor Sketch tool controls

Figure 3.11a Video presented Sketches only (schematic)

Figure 3.11b Video presented Sketches only (screen capture)

To achieve the video mixed configuration the PC output was altered accordingly using a SCAN converter (not shown in the diagram) before being passed to the video mixer unit. In the case of the projection configuration a video camera above the Worker‟s desk picked up the projected MS Paint interface and presented this along with whatever else was in the task space on the Helper‟s TV. In the video window configuration both collaborating parties saw exactly the same video mixed images of live video feed of the task space and MS Paint interface, presented on their TV‟s. In both cases the MS Paint interface was presented to the Helper with enough resolution to enable them to manipulate the required controls to use the paint package with the tablet and pen (by guiding their actions as per feedback from their TV monitor – this is in essence exactly how the Wacom tablet is used in a standard PC / VDU set-up). During

Video Camera Projector Worker Helper TV TV Mixer Whiteboard PC Wacom tablet Worker‟s Hands Assembly pieces Last Sketch Cursor

use, and as a further aid, a prompt sheet (with an enlarged image of the interface) was given to the Helper, with a series of common interface manipulations highlighted (see appendix 3.1). With the Sketch only configurations another system design consideration presented itself, namely the removal of sketch information from the workspace. Previous work has demonstrated the potential benefits of automatic remote sketch erasure (Fussell et al 2004). Whilst installation of an auto-wipe system for the Sketch only system would have been possible, implementing it in the low-tech prototype Hands and sketch system would have had significant design implications. A design decision was therefore made that a manual wipe should be adopted for the Sketch only configuration to keep consistency between the features of the various systems for experimental comparison. Therefore removal of sketches during the use of the Hand and sketch configurations was achieved by manually wiping the desk surface, and in the Sketch only configurations this was achieved by using the digital pen to highlight an area for removal and then using a button on the pen to select a „clear‟ function. Pilot-testing did demonstrate that there was still a difference in the relative ease of use of the two deletion methods. Acknowledging that previous research had suggested method of deletion to be of possible interest it was felt that should experimental work reveal a significant performance difference between the Hands and sketch and Sketch only configurations, the system could be further re-configured to facilitate specific comparison of the relative effects of auto-wiping versus manual erasure techniques.