Existing Conceptual Frameworks

Essentially, Beaudouin-Lafon described two levels for analysis and design of inter- action between a digital system and a person [Beaudouin-Lafon, 2004]: Interaction

paradigmsprovide a user centered high-level conception of the phenomenon of in- teraction. For instance, Reality based interaction, [Jacob et al., 2008], describes in- teraction at a high layer of abstraction using concepts of the physical world; essen- tially a specialized view on interaction between people and digital systems designed to conceptualize interaction techniques. In contrast to this, interaction models of- fer operational descriptions of the course of interaction. Here the interaction itself can be modeled and mapped to specific user actions and system responses. For in- stance, instrumental interaction, [Beaudouin-Lafon, 2000], or direct manipulation, [Shneiderman, 1983], describe the process of interaction between a user and a digital system and the concepts used to compose specific interaction techniques.

A conceptual framework for PPI and mPPI thereby couples these two aspects. It aims at describing interaction between a person and a digital system by means of pen and paper, while at the same time offering a formalized, operational description of the course of interaction, e.g., as required for toolkit design.

Traditionally, conceptual frameworks and models of interaction range from low level, formal human processor models, e.g., GOMS or KLM, [Card et al., 2000], to abstract descriptions, e.g., Reality-based interaction, [Jacob et al., 2008]. As for PPI, interaction with pen, paper and a digital device forms a subset of tangible interaction for which conceptual frameworks of interaction exist, e.g., TAC, [Shaer et al., 2004]. Expressing PPI with these models, however, is cumbersome as primitives relevant for PPI, e.g., gesture and Pidgets, must be constructed out of generic primitives for tangible interaction in general. Hence its vocabulary does not completely fit the do- main of PPI and does not lend itself to support PPI toolkit design without introducing additional complexity.

Therefore, in order to support toolkit design, as elaborated in section 2.4.2, a con- ceptual framework needs to fit the domain as closely as possible, e.g., to provide an adequate vocabulary (R3.1) while at the same time offering machine understandable execution semantics (R3.4). Thus, only conceptual frameworks for the domain of PPI / mPPI provide promising candidates.

In the domain of PPI, only two conceptual frameworks or interaction models have been described so far: RSL, [Signer and Norrie, 2007a], and Steimle’s conceptual framework [Steimle, 2009a].

Resource-Selector-Locator (RSL)

The Resource-Selector-Link (RSL) model proposed by Signer and Norrie has been used to model PPI as theoretical underpinning of the iServer and iPaper framework, [Norrie et al., 2006a]. RSL essentially describes a hyper-document system allowing to link between various resources, both digital and physical. Thereby, it defines links between different types of resources, where selectors specify which particular part of a resource is the source or target of a given link [Signer and Norrie, 2007a]. In the context of PPI, paper artifacts are modeled as resources linking to digital functional- ity. Selectors specify which part of the paper document links to which functionality. Interaction is modeled thereby exclusively as the invocation of links, i.e., following a certain link triggers a system response.

RSLcan be used to describe a broad range of different cross-media links and pro- vides machine understandable expressions (R3.4). It also allows chaining of expres- sions, as links may refer to other links (R3.2). However, RSL does not explicitly model the interaction between user and system and does not offer an interaction vo- cabulary (R3.1). It focuses on the invocation of functionality alone, i.e., triggering

system responses. It also does not support openness and extensibility on the concep- tual framework level (R3.3)37.

Furthermore, RSL models interaction through selectors specifying a particular part of a resource. This limits the range of interaction vocabulary that can be supported at a conceptual level to Pidget and Cross-media link classes; as those closely follow the hyperlink model and location on paper artifacts forms an essential part of interaction. More sophisticated interaction techniques cannot be expressed without extending the model, e.g., gesture based techniques or approaches involving interactive system re- sponses require additional concepts.

To deal with these conceptual problems, the selector semantics have been extended by the authors to allow for defining more complex interaction techniques, e.g., in the

iGesturesystem on top of iPaper / iServer, [Signer et al., 2007b]. This corresponds to strategies used in modern Web applications, where JavaScript frameworks and AJAX are used to work around the hyper-document system nature of the Web. As a result, however, it introduces additional complexity to the design of toolkits, while at the same time lacking concrete domain vocabulary for conceptual modeling.

Steimle’s Conceptual Model

To aid the design of interaction techniques employed in PPUIs (c.f., definition on page 2), Steimle proposed a conceptual framework grounded on empirical research [Steimle, 2009a]. It focuses on answering the questions which interaction techniques are available and which are appropriate in the chosen setting of collaborative knowl- edge work (this aspect is discussed in section 2.5.2). Additionally it provides a struc- turing of the design space that can be used to describe and model PPI.

Steimle’s framework consists of a syntactic layer of core interactions and a se- mantic layer of conceptual activities. Interaction techniques are combinations of core interactions to perform conceptual activities. Described core activities include inking, clicking, moving, altering shape, combining and associating paper artifacts. Concep- tual activities are functionality offered by the system, e.g., annotating, linking or tag- ging. This relates to the triggering of functionality as used in the RSL model described above, but extends it by a domain vocabulary. Both, core interactions and conceptual activities, were derived by observing users in various collaborative knowledge work tasks.

This structuring covers part of the design space for PPI and provides a basic vocab- ulary aiding its exploration (R3.1). Chaining of interaction techniques is an integral part of the framework and thus supported at the conceptual level (R3.2). However,

37

although of course resulting toolkits can be extended through various selectors, links and supported resources

Requirement RSL Steimle’

s

Frame

w

ork

Suitability for Toolkit

R3.1 Design Vocabulary - (x)

R3.2 Composition x x

R3.3 Openness and Extensibility - - R3.4 Machine Understandable x - Table 2.2: Suitability of Conceptual Frameworks of PPI for Toolkit Design

the vocabulary is highly influenced by the domain of knowledge work, which sec- tion 2.2 shows to be only a sub portion of the PPI design space. For instance, how would gestures be expressed by the framework? The closest match is inking, which would also refer to annotations made on paper, or even selections. Therefore the presented vocabulary can be considered as specialized. Also, the framework does not support openness and extensibility, as core interactions and conceptual activities are fixed (R3.3) Furthermore, it lacks formal semantics required for a machine under- standable representation that is required to enable it to serve as foundation for a toolkit (R3.4).