Many computing applications involve several fundamentally different operations like 2D pointing, 3D rotation, text entry, or (on the more complex side) volumetric selections. Just as mouse and keyboard nicely support combinations of 2D pointing and text entry tasks, further dedicated tools could be provided. The diversity of in- terface prototypes in research supports this idea. The combination of complementary tools can enable more complex actions with less individual effort from more involved actors.
Craftspeople have always equipped their workshops with a huge number of differ- ent tools. The enormous quantity of items we find in a common workshop is only partly related to different operations, but every type of tool must be available at var- ious scales (Figure 1.1). Digital tools offer dynamic scaling and related adaptation techniques to adjust these properties on demand, hence a smaller number of devices appears to be necessary. In computer applications, one physical input device (e.g. the mouse) commonly serves for various tasks with exchangeable digital tool tips like pencils, brushes, or selection pointers. On the digital side of this amalgamation, application designers keep adding novel modalities to ever growing tool bars. The comparison with traditional workplaces, however, also indicates that the quantity of modes and digital widgets can be reduced, if the required functionalities can be achieved through the combination of multiple simultaneous input facilities. For ex-
Specialization and Combination of Tools 5 ample, drawing a straight line does not necessarily require another type of tool than drawing freeform shapes, but, the same pencil can be used with or without a ruler.
Figure 1.1: Real workplaces are often crowded with tools. They can be used concurrently and they afford meaningful combinations for extended function- ality.
The effects of tool combinations are limited by physical constraints in traditional workshops. Computer applications, on the other hand, can potentially offer mean- ingful interpretations for any possible combination of inputs. Hinckley et al., for example, recently demonstrated an encouraging collection of functionalities that can be realized through such combined input of touch and pen [143]. The specialization of tools and their use in combination are fundamental principles of interaction de- sign. Examples can be found everywhere, not least, in the realm of current computer interfaces. The following sections introduce some of the latter.
1.1.1
Task-Specific Tools
Based on an analysis of the benefits and drawbacks of various interface technologies and interaction techniques, we can choose the most suitable ones for the different re- quirements of text entry, object rotations, color adjustments, and many other tasks. The more we narrow down the specifications of the task, the more we can optimize for best performance. Practical applications, however, involve several such tasks with varying requirements. There are two approaches to deal with this diversity of de- mands. A generic interaction tool can be provided that serves the majority of tasks reasonably well or highly specialized instruments can be devised for each of them. The optimal balance between universality and differentiation always depends on the context of application.
Besides task correspondence, the general user performance in the manipulation of various interaction tools must also be considered. Direct manipulation seems to be most efficient if it is bound to physical support surfaces for motion input (see Chapter 2). Direct 3D manipulation of virtual objects lacks such physical support. Therefore, it is not surprising that 2D user interfaces are by far more established than their 3D counterparts – even for the specification of 3D transformations. Whether the performance benefits of 2D input outweigh the drawback of more indirect in- put mappings is open to debate and further research. Proponents of 3D input de- vices often argue with the integral operation of related attributes, benefits of propri- oceptive feedback and the direct kinesthetic correspondence to the interaction task (e.g. [138, 231, 281, 329, 357]), while critique is often centered on lower input accuracy and higher fatigue (e.g. [31, 335]).
Be that as it may, people tend to move simultaneously through multiple dimensions and attribute spaces during coarse target approximation [21,131,190,242,356,357,381]. During fine grained parameter adjustments, instead, different degrees of freedom are often operated subsequently [21, 250, 251, 347]. For 3D object manipulation, the pre- dominant 2D input paradigm can be considered a middle ground between both de- mands. However, it neither supports unconstrained direct manipulation for coarse approximation of 3D targets, nor does it offer implicit constraints to a single dimen- sion for accurate placement. On-screen widgets are therefore used to further reduce the input to only a single axis.
1.1.2
Interface Adaptation
User interfaces can also be adapted dynamically to changing requirements, either implicitly or explicitly. The most successful examples of the first approach are adap- tations of the transfer function based on motion velocity (e.g. [97,242]) and automatic object snapping to potential targets or related constraints (e.g. [25, 36, 37]). Intuitively this approach seems to offer the most potential if the adaptation builds on raw user input with many simultaneously operated degrees of freedom and applies reduction only when necessary towards the end of a placement task.
Alternatively, users can explicitly adapt parameters of the transfer function or chose among tools with different characteristics. The accessible presentation of multiple tools and settings, however, occupies valuable interaction space – either physical or virtual. Moreover, the required choice among multiple options can be detrimental as described by Hick’s Law [132] and switching between them takes time. These issues can be more or less pronounced, depending on the type of involved tools and their arrangement. If not designed properly, the drawbacks can impair the benefits of ded- icated task suitability. Ideally, the choice of tools, modes, and settings could be spec- ified implicitly while focusing on an uninterrupted manipulation process. In future, brain-computer interfaces could be used to realize such implicit mode changes [320]
Specialization and Combination of Tools 7
1.1.3
Virtual Interaction Widgets
Most computer interfaces offer only few physical input options, and provide addi- tional functionality with on-screen widgets. These virtual tools can be operated with the same physical input. They can be displayed and arranged dynamically in rela- tion to the operable parameters of selected objects, which facilitates the users’ choice among them. Widgets can be used, for example, to translate 2D motion input from a pointer in screen space to other attribute spaces of an application, e.g., color or 3D position. The mapping often involves a reduction of the two-dimensional input to a single parameter, e.g. via a slider widget. Also the aforementioned mapping of 2D in- put to 3D object manipulations is often realized with on-screen widgets that support subsequent transformations on individual axes (see [302] for an overview).
Widgets are graphic representations of their functionalities, which makes them very versatile and comprehensible. Unfortunately, the available degrees of freedom can only be reduced and not increased. Tasks that involve multiple degrees of freedom must be operated in multiple steps. Consider the assembly of a complex 3D object from multiple parts using a mouse with digital manipulation widgets. For each ob- ject manipulation, users must operate translations and rotations along three spatial dimensions subsequently – each time involving the acquisition of the corresponding handle. It seems unlikely that this is the most effective interaction method.
1.1.4
Specialized Input Hardware
The provision of multiple specialized physical input devices can be beneficial for several reasons. Most importantly, the shape and weight of physical devices affords different types of action. For example, 2D input can be realized with direct touch, a pen, or a mouse device. Drawings require such 2D input, but the results differ strongly between the three technologies [373]. Furthermore, the design of physical input devices can accurately fit the type and the number of parameters that are to be operated integrally. Last but not least, physical devices support adjustments in various attribute spaces based on tactile and kinesthetic feedback. As a consequence, physical tools may require less visual monitoring for their operation than virtual in- teraction widgets [188, 362]. Users can even operate multiple physical input devices without losing their focus on higher-level aspects of the interaction task [137, 342] As mentioned above, the simultaneous availability of multiple input devices affords their combination in various meaningful ways to achieve additional functionalities. Depending on the task at hand, their individual capabilities can be constrained or ex- tended. The most established combination of computer interfaces is probably that of mouse and keyboard. The 2D pointing device enables direct manipulation while the keyboard provides symbolic input. Good interaction design exploits both interfaces for more efficient operation. We know several common patterns of their complemen-
tary use. When placing objects on a virtual canvas, for example, coarse approxima- tion is rapidly achieved with a pointing device, but the direction keys of the keyboard facilitate fine-grained adjustments on individual axes.