Syntax: Core Interactions

The syntactic level encompasses interaction primitives of of PPUIs. These core in- teractions are independent of the functionality of the PPUI. Instead, they model in a generic way how users interact with pen and paper.

Empirical grounding

The core interactions are based on empirical evidence. We therefore briefly summa- rize the observations we made in an ethnographic field study on how users work with pens and printed documents. Following our observations, syntactic interac- tions with pen and paper can be divided into the following three categories:

1. Writing: Writing on an individual sheet of paper (e.g. annotations, handwrit- ten links and keywords or symbols for tags). Participants often partitioned

4.2 Interactions 93

the available space into separate functionality zones, e.g. reserving the left or right margins of the document for keywords in order to provide for a quick overview on all tags on a page.

2. Moving, Arranging, Modifying: Leveraging the material aspect of paper sheets, which can be flexibly arranged in the physical space. Specific spatial combina- tions of two or more paper sheets convey semantics (e.g. relating documents by putting them into a folder or on a stack or marking important pages with bookmark stickers). Moreover, the shape of physical paper sheets can be mod- ified, e.g. by bending, folding and tearing.

3. Pointing: In the co-located collaboration of learning groups, we frequently ob- served pointing gestures to documents. Very often participants did not only point to one single document but consecutively to several documents or pages in order to express relations between the contents they pointed to.

Core Interactions

These results lead to the following core interactions.

Definition 9(Core Interaction). A core interaction is defined as an operation that a user performs by manipulating one or more page areas using a digital pen or her hands.

Examples of page areas comprise a printed document page, a printed button el- ement or an adhesive bookmarking sticker. First and foremost, page areas are con- tained on printed sheets of paper. However, by analogy, areas which are displayed on a display can equally act as page areas.

The operations performed with these page areas are the following:

• Inking: Writing with the digital pen on a page area. This includes free form handwritings and drawings that are digitally captured. Moreover, specific symbols and pen gestures may be performed to issue a command.

• Clicking: Performing one or more pen taps on a paper area to issue a command (e.g. on a printed “button” area). This is inspired by traditional pointing ges- tures. We distinguish clicking from inking for the following reason: While inking leaves visible pen traces and permanently alters the document, click- ing is volatile, leaving it conceptually unchanged.

• Moving: Changing the physical location of the page area. This also includes picking it up and putting it down as well as flipping pages.

• Altering Shape: Altering the physical shape of a page area, for example by bending, folding or tearing it.

An important characteristic of paper is that it affords using several sheets of paper at a time. The following core interactions comprise the use of two or more sheets:

• Combining: Creating or modifying arrangements of page areas. This may be rather volatile (e.g. paper sheets laid out on a desk) or rather permanent (e.g. attached paper stickers, documents filed in a folder).

• Bridging: In contrast to physical combinations, bridging is a logical combina- tion of several areas. This complements physical combinations or substitutes them when these are impractical or impossible. Inspired by consecutive point- ing on several items, we model bridging as a connecting pen gesture on two areas.

The same core interaction may have a different meaning if it is performed on a different type of page area or using a different tool, e.g. a digital eraser.

Discussion

We deliberately model combining and bridging as separate core interactions instead of combinations of inking, clicking and moving. The reason is that one of the most important affordances of paper is to support two-handed interaction and hence a really concurrent use of multiple sheets of paper.

Table 4.1 depicts how these core interactions are inspired by interactions with tra- ditional paper and how they correspond to interactions within Graphical User In- terfaces. As GUIs incorporate metaphors of traditional desks, they offer somewhat equivalent interactions for inking, clicking and moving. In contrast, traditional GUIs do not support two-handed input but only one single focus point. Consequen- tially, combining and bridging should be rather considered as sequential manipula- tions of individual areas. Moreover, the interaction of altering the shape leverages the physical specifics of paper environments and goes beyond what is possible in GUIs.

PPUIs should account for the rich interactions that are possible with paper. They should use a broad spectrum of core interactions. In Section 2.2, we have seen that paper affords the flexible spatial organization and the concurrent use of multiple sheets of paper. PPUIs should leverage this affordance and support the core inter- actions of combining and bridging. This stands in contrast to a design which is inspired by the GUI paradigm, in which interaction is restricted to single points of focus (due to the single mouse pointer).

Our interaction techniques to be presented in the following chapters rely on four core interactions. Instead of moving, we support combining, as cognitive activities typically consist of creating or modifying arrangements of several sheets but not of moving a single sheet without relation to others. Further, we very rarely observed manipulations of the physical shape of paper sheets in learning at universities. As moreover the tracking technology for detecting these manipulations reduces the mobility of paper use, this core interaction is not part of our design.

4.2 Interactions 95

Traditional Paper

PPUI GUI (following [BL00])

Single sheet

Writing Inking Text entry Pointing Clicking Pointing/Clicking

Moving Moving Dragging

Altering shape (bend- ing, folding, tearing) Altering shape - *

Multiple sheets Arranging and Combining

Combining - * Subsequent

pointing

Bridging - *

* No core interaction (performed by combining several core interactions)

Table 4.1: Comparison of Core Interactions