Image Loading

For sharing new photos, we need a way to indicate a set of new image files to load onto the interface. The intention is not to use a file selection dialogue from a traditional interface – this would go against our guidelines. Another student has developed techniques for associative search for loading files onto the table using a Cruiser plugin (see §6.3), which provides search access to files on a remote computer. However, our scenario (§1.1.2) deals with unsorted digital photographs, which are difficult to search, and behaviours for searching and sharing photographs are often integrated [Frohlich et al., 2008]. PhoTable supports this by using intelligent browsing techniques, but the first step is presenting the set of images to browse.

The scenario presented in the Introduction (§1.1.2) gives us a use case we should support. Someone returning from a holiday, having taken a large number of digital photographs, will have a memory card full of images that they wish to share, and possibly annotate. The simplest answer is simply to plug in the digital camera (via a cable) or memory card (into a card reader) and have the photos appear, ready for sharing. And this is exactly what is done – the Cruiser framework includes a plugin that requests these events from the operating system and immediately searches newly inserted storage devices (which include cameras) for loadable images, possibly with associated audio recorded by the camera.

This extends also to other removable storage, such as USBflash thumb-drives. So, for example, inserting a USB drive will load the images on it. However, in this case the photos may already be organised, requiring an alternative browsing technique, which we discuss next. Browsing large unsorted collections of many hundreds of photos requires a different browsing approach, discussed in Section4.5.5.

4.3.7 Browsing

A Browser is an object that groups a number of photographs, allowing them to be browsed. A Browser holding nine photographs is shown in Figure 4.5. One, simpler, type of Browser

4.3. High-Level Functions CHAPTER 4. USER VIEW

has its contents derived from a folder on disk. This may be a folder on an internal hard drive9, or the root folder of newly inserted removable storage.

Essentially, we needed a way for users to interact with a hierarchical file system and we do this via a Browser object. Whenever a Browser is created, Cruiser searches for a sub- folder containing images and creates low-resolution thumbnails for them. The thumbnails are attached to the front of the Browser, laid out inside a margin using space-filling techniques [Cockburn et al., 2006]. A margin allows the Browser to be moved around, rotated and resized; carrying all its thumbnails with it.

When any thumbnail is dragged off the Browser, Cruiser makes a copy; leaving the original in the Browser for future copies. At this point, Cruiser also upgrades the quality of the image. Thus, any photograph on a Browser object can be copied, and the copy placed at the desired position, with a single stroke.

If multiple sub-folders are found to contain images, each sub-folder is loaded into the Browser as a SubBrowser. The first image found in each sub-folder (or a specially named

folder.png image) is chosen as the SubBrowser background. SubBrowsers are arranged like

thumbnails, but their corners coloured like Browsers. Dragging off the SubBrowser creates a new Browser, using the corresponding sub-folder as the new root of the Browser, and immediately loading thumbnails (or any further sub-folders) found.

This action is shown in Figure 4.6. Different colours indicate the active selection (red) and the newly created object (white). Once the action is completed, and the new Browser released, the SubBrowser returns to its original position in the original Browser. Dragging off thumbnails from SubBrowsers works similarly but triggers loading of the high-quality image (rather than image thumbnails).

Note that the colours here are chosen to optimise contrast. We cannot anticipate the contents of images in order that we may chose more subtle colours – they might not be easily distinguishable on all images that would be loaded on the interface. In nature photography, blue (skies and water) and greenery are common, so reds and yellows were our first choice for maximum contrast. Future work may include investigating a colour scheme, or textured borders, that may be more aesthetically pleasing.

4.3.8 Copy

The collaborative nature of tabletop interaction means that we can expect multiple users to need to speak about, look at and act on the same object (see also §3.3.4). Such an interaction possibility is suggested for ConnecTables [Tandler et al., 2001] where “a simple gesture creates a simultaneous view of an information object,” but this interaction was not explored further. A copy was also suggested in one usage scenario for the UbiTable [Shen et al., 2003a], when moving a document into one’s personal area (to create a copy on their laptop), but a copy operation was not explicitly provided.

The previous subsection introduced one example of copy in Cruiser. Thumbnails (and SubBrowsers) are not removed from the Browser that holds them. Instead a new copy is created. For images, the new copy will be of higher resolution. For SubBrowsers, the copy is now a full Browser, and the contents of that sub-folder are loaded into it. Copy is a key element of our interaction design (§3.3), so it has been explored further.

4.3.8.1 Brainstorming possibilities for Copy

We first considered possibilities at conventional interfaces: a menu bar, context menu, or keyboard sequence. These all pose problems on a multi-user tabletop interface. For example, any display of menu text is prone to orientation and legibility problems; context

9_{This path may be pre-configured or passed as a command line argument, e.g. by dragging the folder}

The line – – shows the path of the user’s touch point from start to finish (where the hand icon appears); line and hand do not appear on the interface (users interact directly and so no cursor is required). As the cluster is dragged off the parent Browser, a copy is made (a “SubBrowser” in the lower right), and loading of imagethumbnails within the cluster begins immediately.

Figure 4.6: The Browser Interface with 7 SubBrowsers, as one is dragged off

menus are a problem with the limited input primitives. These and other problems stem from the fundamental differences of tabletop interaction.

Drawing instead from our guidelines (§3.1), we explored the idea of a special object that had the sole purpose of copying. One needs to be careful about having many such special objects at a table but it is an approach which deserves some exploration.

We also explored the potential for existing elements of our interface to support copying. This provided two additional approaches: copy by dragging an item off the file browser; and make use of the Capture Frame, which creates new images by grabbing all the visible content under it.

Our design approach produced three special interactive objects which can copy. Aside from copying, they have the behaviour that we have established as fundamental to objects in our framework (i.e. move, rotate, resize):

• The Frame (§4.3.2), • The Browser (§4.3.7), and • The PhotoCopier.

4.3.8.2 PhotoCopier Object

This interface object copies photographs. It has a stylised background with an image of an office photocopier to indicate its function. In addition, the corners indicating the rosize hot spots are drawn in solid colour (rather than transparent), to distinguish it from regular photographs. This appearance is similar to the Browser, but the background is fixed, and

4.3. High-Level Functions CHAPTER 4. USER VIEW

Figure 4.7: The initial layout shown to participants in our copy study

it only ever holds a single child image, attached as follows. A PhotoCopier object can be seen left of the centre in Figure4.7.

Any object (e.g. a photograph) moved over the PhotoCopier object acts as though it has been copied into it, replacing any existing object. The action to do this is a swipe (§4.3.4.1) – an uninterrupted move – which causes the initial copy to appear attached to the front of the PhotoCopier. The PhotoCopier can be moved, rotated and resized like other objects with the copy remaining attached. When any user then tries to move the attached copy off the PhotoCopier, we instead make a another copy, which immediately becomes that user’s active selection. Again, the move operation is not interrupted. Thus the process of selecting the (original) image to copy, initiating the copy and placing the copied object occurs with two strokes. Subsequent copies of the same object require only the placement stroke.

Still photographs are not the only interface objects that may be copied. For example, the Capture Frame may be swiped over the PhotoCopier. Dragging the Frame off will the create duplicate Capture Frames that retain all the properties and behaviours of the original. For a single user, or in storytelling, it is not normally necessary to have multiple frames. However, the PhotoCopier’s behaviour extends to most objects on the interface, including storage bins, described next, for which it is often useful to have duplicates. 4.3.8.3 Comparing Possibilities

Using a combination of an informal “keystroke” analysis (see §3.2.1), heuristic evaluation (§4.1.1), user trials (formal and informal), and a questionnaire, we performed a comparison of our three ways to copy objects in Cruiser [Apted and Kay, 2006a]. The Browser requires the least numbers of “keystrokes”, as something is merely dragged off to create a copy. Applying heuristics suggest the PhotoCopier may be superior because it draws closely on analogy from the real world (N2), has a recognisable action (N6), and has a clearly defined purpose (G1d).

In formal trials we asked seven participants to “make six copies of any photograph as quickly as possible,” after providing a 5-minute interactive tutorial to familiarise and allow them to experiment with the techniques. Participants were provided with the initial layout of objects shown in Figure 4.7. The task was lightweight as we wanted to examine the participants’ natural reactions and recollection of the techniques available to them. We wanted the participants to choose the copy technique they thought made the most sense and were comfortable using, even after such a brief introduction to the interface.

All participants were able to complete the task without assistance. However, contrary to what the stroke analysis would indicate as fastest (the Browser), we found that 5 of our 7 participants first chose an image from the Browser; dragged it off, then over

the PhotoCopier; then dragged 6 copies off the PhotoCopier. When asked, most users commented that they felt that each object had a specific task – the Browser for selecting photographs, and the PhotoCopier for copying photographs – even though the objects were explicitly introduced as three different ways to copy photographs.

Some users commented that they particularly liked the analogy that the PhotoCopier provided and found it very easy to use. Some mentioned that the Frame seemed complicated and over-powered for this task, so they did not use it. Three of our participants were particularly excited by the prospects the tabletop interface offered, and continued to experiment with the interface, long after the questionnaire.

See Apted and Kay [2006b] for more details of this study.

4.3.9 Storage Bins

Our storage bins draw on the ideas of Scott et al. [2005] – the bins themselves are movable and resizable, and adding an item to the storage bin triggers an animation as it is placed within. However, in part due to difficulties Scott et al. observed their participants having, and our own desire to avoid menus in our system, move and resize operations on the bin are not activated through a radial context menu as for Scott’s bins.

Instead, spacing is maintained around images in our storage bins. Selecting the bin rather than an object within it will move the bin (and all objects within it). Similarly, and as for regular images, selecting a corner of the bin will initiate the rosize action. Thus, our bins are also rotatable, whereby all contents within the bin will also be rotated as if glued on. This contrasts Hinrichs et al. [2005a], which has an implementation of a lazy Susan type rotation directly upon Scott’s storage bins, whence they became interface currents; usable as a storage area. Note that this storage bin move, rotate and resize behaviour is consistent with all other objects in the Cruiser interface, thus satisfying our consistency guideline.

One problem addressed by storage bins is the issue of clutter. Reducing clutter is one of our guidelines, and storage bins help by providing a storage space of infinite capacity that uses a fixed amount of space. This is achieved by using space-filling techniques10[Cockburn et al., 2006]. The same technique is used to represent folders containing arbitrary numbers of images in Browsers (4.3.7). Furthermore, the space used by these storage bins is resizable and movable, around the interface.

Storage bins provide another mechanism that works in concert with the Browser to alleviate the clutter problem. Dragging new images off the Browser creates copies, which can quickly clutter the interface. So, when a new SubBrowser is dragged off its parent, other SubBrowsers and thumbnails copied from the same parent are automatically cleaned up – removed from the interface. However, any image placed into a storage bin (or attached elsewhere) will remain on the interface. Thus, images that would otherwise clutter the interface are instead placed within a well-organised storage bin. New, empty storage bins may be generated after swiping any existing storage bin over the PhotoCopier object.

The storage bin was designed for users to collect a set of related photographs. For example, if they needed to explore several photographs in their collection to select just the ones for a particular story, they might place each selected image in a bin while they continue exploring and selecting other photographs. Study participants have also commented that the storage bin is a good tool to place photos that one might want to print, for example.

4.4. Extended Functionality CHAPTER 4. USER VIEW

4.4

Extended Functionality

The Cruiser framework has been designed to encourage code reuse and easy experimentation, making it easy to explore new and useful functionality as an extension of existing techniques. Section 3.4 motivated some additional functionality for tabletop interfaces that it is worthwhile to explore. Cruiser facilitates this exploration with plugins that are packaged with the Cruiser framework for reuse by application designers.

The interface elements described in this section contribute new functionality that is not used by the PhoTable application, but demonstrate the flexibility of the Cruiser framework. They also provide elements that can be reused for new tabletop applications, and have been used in numerous demonstrations of the interface.

This section covers only plugins implemented by myself. The Cruiser framework was written to be used by others and, in this way, has facilitated the implementation of further extended functionality created by other researchers. These extensions are also implemented as plugins, and are discussed in §6.2.

4.4.1 Interactive Images

An Interactive Image is simply an Image object whose texturecan be updated dynamically. In the current implementation, updates are sent to the application from a remote computer using the VNC protocol, which is well suited to representing the changes that occur on a desktop computer display. It is not well suited to video, for example, but for typing, static updates, cursor and window movements it is especially good. For example, a PowerPoint presentation could readily be supported inside an interactive image. The display is held within an object that behaves like other images and is able to be moved, rotated, resized, attached, captured and copied. An example interactive image can seen in Figure4.2 on page 68, indicated _Â.

To support this, a VNCclient has been written, which talks to any compliant VNC

server (e.g. RealVNC, TightVNC, X11vnc, etc.), running on any operating system. The remote display may be sourced from another computer accessible via any network, or a

virtual display running on the same computer. Multiple servers can be communicated with

simultaneously, with each screen appearing in its own, interactive image. We support the

VNCpassword authentication method – the password is saved in the configuration file and authentication is by a 3Data Encryption Standard (DES) challenge-response mechanism. We currently support raw, copy rectangle, Rise and Run-length Encoding (RRE) and CoRRE update mechanisms, which are encodings used by the VNCprotocol to efficiently update only the altered parts of the remote display.

After establishing the connection and authenticating, we wait for framebuffer updates. When an update arrives, the interactive image updates a hidden framebuffer held insystem memory, collecting the bounding box of all sub-updates. When the update is complete (we only get one update until we ask the server for another), we generate a sequence of texture updates and notify the main thread when they are ready. Up until now, this all has occurred in a background thread to keep the main interface responsive.

At a suitable point of low activity in the main thread, the parts of the image texture that need updating will be updated by transferring the preprocessed information to the graphics card. The time this takes depends on the size of the updated area. For small updates such as cursor movement or typing new characters it is very quick – usually less than 100 ms.

This process propagates any updates on the remote computer screen to the image in Cruiser. But the image on the tabletop is also interactive. However, to make the image interactive we now need to break some of our guidelines. For keystrokes, the answer is straightforward: introduce a keyboard (preferably wireless) which transmits keystrokes to

the remote display corresponding to the currently selected interactive image. But now we have a keyboard, breaking our pervasive environment guideline (G2f).

For mouse cursor movements, we need a way to distinguish a regular move of the entire image, from a request to interact with the underlying display. This introduces modes, breaking guideline G1f. A number of ways to indicate the mode are available: a button on a stylus may be pressed, a different coloured stylus can be used, or the interactive image may be flipped. Flipped interactive images appear the same (they don’t have a reverse side), but they cannot be moved until flipped over again. In all cases, dragging a touch point over the interactive image will instead perform a click-and-drag on the remote computer at the corresponding location – the position is translated, rotated and scaled so that it corresponds with what the user sees inside the image on the tabletop.

However, breaking our guideline has a negative effect. Users are often confused about the mode of the interactive image when flipped. We write “interactive” over a flipped interactive image, but only when it is not selected, otherwise it would interfere with the interactivity. This means users often inadvertently interact with interactive images when they instead intended to move it. Similarly, the alternative of switching pens often causes users to forget which is the move pen and which is the interact pen. Using a stylus with a button currently works best, but the hardware support needs improvement. Furthermore, when using fingers in a purely touch-based interaction, this option is not available. Improvements on the indication of mode may also help ameliorate the negative effects, but this is left for future work.

4.4.2 Video Images

TheVNC protocol is poorly suited to updating rapidly changing displays, such as those