Mapping Between Syntax and Semantics

Having defined the semantic and the syntactic level of interaction, we now focus on the intersection between both levels. We analyze how syntactic core interactions can be mapped to semantic activities, such as annotating or linking documents, and to their sub-activities. The latter are actions like specifying the first or the second link anchor, defining the scope of a document a link or tag applies to, defining a new tag category or applying a tag category to a document passage. The number of (sub-)activities is larger than the number of core interactions. For this reason, the model must rely on some kind of multiplexing between a smaller number of core interactions and a larger number of semantic activities.

In an influential interaction model, called Instrumental Interaction, Beaudouin- Lafon [BL00] discusses different types of multiplexing for tools in Graphical User

4.2 Interactions 97

Interfaces. He analytically separates tools (which he calls instruments) into a phys- ical and a logical part. While the physical part is the input device, for example the mouse, the logical part is realized in software on a screen, for example the button in a toolbar palette for selecting the instrument for drawing lines. He discusses two types of multiplexing between a single device and many logical instruments: tem- poral and spatial multiplexing. Temporal multiplexing means that the user selects an instrument which remains active until another instrument is selected. This corre- sponds to modes. Spatial multiplexing means that the instrument becomes active if the user positions the input device in an area of the screen which is associated with this instrument.

We adapt and extend Beaudouin-Lafon’s types of multiplexing and distinguish the following five types of multiplexing, which are used to map core interactions to activities. These types do not exclusively apply to PPUIs, but are all possible in GUIs as well. A syntactic interaction that performs a specific activity might use one or more of these types of multiplexing between core interactions and semantic activities.

Compositional multiplexing. A key finding of our field work was that in traditional paper practice, users often combine several core interactions to perform a single conceptual activity. The user may for example attach an adhesive indexing sticker onto a document page and then write a label on this sticker. We call this composi- tional multiplexing and judge it to be – together with spatial multiplexing – most important for interaction in PPUIs. This principle enables users to draw upon a small repository of simple core interactions. These act as flexible building-blocks which are used and reused for multiple conceptual activities.

Figure 4.3 on p. 91 depicts how the model is used for the interaction techniques of CoScribe, which will be presented in the following chapter. CoScribe supports the conceptual activities of annotating, linking and tagging documents as well as of defining the scope of annotations, links and tags. The edges between the upper and the lower level indicate how the semantic activities of our application domain are composed of several simple core interactions.

Spatial multiplexing. In traditional practice of working with paper documents, dif- ferent sheets of paper or different areas on one sheet have different functions. Ac- tions can have different meanings depending on the type of paper medium which is used. For example, writing a label on an index sticker possibly has another meaning than writing the same term on a document page. Spatial multiplexing fits well with working with objects in the physical space. The first reason is that much space is available for different page areas, typically more than on a screen. Second, it is easy and intuitive to select a given area by physically grasping it.

Applied to our model of PPUIs this means that PPUIs should comprise different types of page areas that serve different purposes (e.g. document pages, folders and index stickers). The same core interaction performed on a different type of page area

leads to another result. For example, if the core interaction of inking is performed on an adhesive bookmarking sticker, it might define the label of an index, whereas inking on a document page might create a free-form annotation.

Gestural multiplexing. Different words or gestures written with the same pen on the same type of page area serve different purposes. For example the keyword “important” made in a margin column might be a tag while “cf. page 3” might be a link.

Gestural multiplexing is very powerful, as a large number of gestures can be de- fined. Moreover, it does not require other tools than one pen and unspecific sheets of paper. However, a heavy use of gestural multiplexing runs the risk to create a command-based interface which requires that the user memorizes a large num- ber of possible counterintuitive commands instead of naturally working with doc- uments. For this reason, CoScribe uses only a small set of very simple gestures, such as points, lines and tapping followed by holding down the pen, and puts the emphasis rather on a rich variety of paper tools.

Temporal multiplexing. The meaning of an action can depend on which action the user has made beforehand. This means that the system has modes. For example, several differently colored buttons for choosing ink colors might be printed on a page. By tapping with the pen on a button, the user selects the corresponding color. All subsequent inking interactions use this color until another one is selected.

Temporal multiplexing is critical, as this requires that the user performs an ex- plicit action to activate the mode and moreover the user may become unaware of the currently activated mode. The latter issue is particularly crucial if in paper-only environments no real-time feedback on the current mode of the system can be given. For this reason, the design of CoScribe avoids temporal multiplexing. Only the core interaction for bridging is modal, enabling the user to create two separate end points one after another. But even this interaction exists in a modeless variant where both end points are defined by one single action.

Device multiplexing. If performed with another device, the same actions can have a different signification. For example, inking with a digital pen which has a ball- point tip might be used for making textual annotations, whereas inking with an- other type of digital pen having a highlighter tip might be used for marking pas- sages. Device multiplexing is powerful if the number of devices keeps manageable and if the repartition of functions across devices is clear and intuitive.

CoScribe does not use device multiplexing because one of our goals is to reduce the costs of devices to make digital pen-and-paper interaction not significantly more expensive for the end-user than traditional pen and paper. Moreover, there is empir- ical evidence that students tend to use one single pen rather than switching between many tools [Mar97].