Basic Framework Structure

Based on the requirements reviewed in section 4.1.1, a conceptual framework of PPI was conceived. It enables interaction designers to express precisely which interac- tions are supported by a PPUI. Thereby, interaction is described through elementary aspects of user actions along an open set of conceptual core dimensions of interac- tion. Initially, five core dimensions constitute the framework, hence its nameW5

. Thereby,W5

not only facilitates description of interaction, it is specifically designed to serve as foundation for PPI or mPPI toolkits and thus offers semantics allowing to map conceptual abstractions to system responses.

W5

describes interaction, i.e., PPI or mPPI, between a user and a digital system at the abstraction level of interaction techniques. As such, it allows designers express- ing interaction techniques through formulating expressions. An expression thereby focuses on describing the course of interaction encompassed in an interaction tech- nique through an operative description of input, i.e., atomic, observable aspects of user actions along conceptual dimensions. These observable aspects are called inter-

action predicates. Expressions correspond to composite chains of interaction predi- cates. In addition, expressions support composite chains of interaction predicates and other (sub-)expressions in order to form new expressions. This enables composition of interaction techniques, i.e., an interaction technique can consist of several other interaction techniques (see explanation of composition below).

This approach enables designers to formulate the input part of interaction tech- niques supported by an application. Input can then be bound to function and feedback. InW5

, both aspects are not explicitly modeled as they depend highly on the Hard- ware / Software composition of the targeted application. However,W5

binds function to expressions; that is, whenever the system observes all input as described in an ex- pression, e.g., when an interaction technique was performed by the user, the system

invokesthe associated function. Similarly, it binds feedback to interaction predicates; whenever the system observes a particular atomic aspect of interaction, it triggers feedback (if required).

In summary, the three basic structural constituents of theW5

conceptual framework of PPI are

Expressions Expressions correspond to operative descriptions of the input part of interaction techniques. They consist of chains of elementary aspects of user actions, the so called interaction predicates. Chains consisting of varying com- binations of interaction predicates and other (sub-)chains are possible in order to support composition of interaction techniques.

Interaction Predicates Interaction predicates are atomic, observable aspects of user actions along conceptual dimensions. As explained below, interaction pred-

Figure 4.2: Dimensions of the W5 Framework

icates fully coincide with exactly one dimension. Thereby, multiple interaction predicates can describe a single, elementary user action, e.g., pointing with a pen on a Pidget. As shown below, interaction predicates correspond to the de- sign vocabulary inW5

.

Dimensions Conceptual dimensions structure the design space and serve as con- tainers for interaction predicates. As such, dimensions provide the structural frame of the framework and guide analysis toward an initial design vocabulary; additionally, dimensions can aid interaction designers in conceptualizing inter- action techniques as they span the design space for PPI input.

Core Dimensions

As described above, conceptual dimensions classify interaction predicates. Classi- fication is thereby based on conceptual properties of the interaction predicates, i.e., conceptual properties of atomic, observable aspects of user actions with a digital pen. Here it can be observed, that in PPI based systems user actions can be classified ac- cording to five central dimensions as illustrated in Fig. 4.2. These five dimensions constitute the core dimensions of the W5

conceptual framework and lend its name. The five dimensions are:

W1 Where?: Spatial dimension

W2 When?: Temporal dimension

W3 What?: Content dimension

W4 Why?: Contextual dimension

All interaction predicates fully coincide with one of these dimensions. Spatial pred- icates (Where) describe aspects of the location of user actions, e.g., where the user touches the paper with a pen. Temporal predicates (When) describe aspects of the timing of user actions, e.g., when the user touches the paper with a pen. Predicates of the content dimension (What) refer to aspects of digital ink contents, e.g., what the user draws or writes with the pen on paper. Gestures or written commands belong to this dimension.

These first three dimensions directly relate to digital ink, i.e., the data observable by a PPI based system without additional information. However, as argued by Steimle, the perspective on the information ecology and the semantic dimension of interac- tion should not be neglected, [Steimle, 2009a]. Therefore,W5

additionally encom- passes the contextual dimension describing associated purpose of actions (Why), e.g., currently executed tasks, and the originator dimension describing associated actors (Who), e.g., users. These five dimensions serve as core dimensions spanning the de- sign space. However, completeness is not assumed as argued below, i.e.,W5

allows adding conceptual dimensions where required.

Examples. In order to illustrate how core dimensions structure the design space,

consider occurrence as a simple spatial interaction predicate2. This interaction predi- cate describes where user actions occur through specifying whether system input can be observed at a certain place, e.g., when touching a certain paper region with the pen. As such, it can be directly used as an interaction technique triggering appli- cation functionality, i.e., as Pidget as in PaperPoint, [Signer and Norrie, 2007b], or

NiceBook, [Brandl et al., 2010] (c.f., section 2.4.1).

Similarly, a simple interaction predicate from the temporal dimension could de- scribe user action observed at a certain point in time, e.g., when the user has to touch the paper with the pen at a certain time. This might be used in a voting system, where the user has to mark a box with an X at the same time the desired choice (out of many) is shown on a screen. An example for a content predicate would be the aforemen- tioned gesture predicate, e.g., when the system observes a checkmark gesture. Similar examples describe the originator and contextual dimensions. For instance, depending on the originator an application might, e.g., accept or reject a command. Further- more, the current task can also influence available functionality, e.g., enabling modal interaction in PPUIs.

When forming expressions in order to describe interaction techniques, these predi- cates can be used standalone, as in the examples above, or predicates can be combined.

2_{This corresponds to the At}

Rpredicate derived on page 157 in section 4.2. Here this and the following

exemplary predicates are used only to illustrate the concept, not as a predefined constituent of the empirically derived basic domain vocabulary.

Combined predicates thereby reflect on multiple aspects of user actions along several dimensions. Additionally, combined predicates can express sequences and composite expressions as discussed below. For instance, an expression combining the spatial and temporal predicates above can express an interaction techniques where input has to be observed at a certain place at a certain time. In this case, application functionality would only be invoked if the user taps a certain paper region at a certain point in time.

Fundamental Concepts As discussed above,W5

allows forming expressions in order to describe input of in- teraction techniques through interaction predicates. Interaction predicates stemming from the five core dimensions introduced in the last section represent atomic, observ- able aspects of user actions. As the constituents of expressions, interaction predicates correspond to the domain vocabulary inW5

.

Domain Vocabulary. InW5

, there is no fixed domain vocabulary, i.e., no fixed set of interaction predicates that constitute the framework and claim to span the entire design space. Instead, W5

applies the principle of openness as discussed below to both, dimensions and interaction predicates (i.e., the domain vocabulary) yielding an extensible set of core dimensions and core predicates. However, in contrast to core dimensions that are defined based on conceptual properties of interaction, core pred- icates are empirically derived from representatives of the three essential classes of PPI techniques laid out in chapter 2, section 2.4.1 (c.f., section 4.2). W5

takes this approach in order to ensure a minimal, relevant basis interaction predicates.

Invocation. As laid out above,W5

enables designers to formulate expressions de- scribing the input of interaction techniques. Input is bound to functionality and feed- back using the principle of invocation: whenever a single aspect of a user action (i.e., interaction predicate) contained in an expression occurs, the system invokes associ- ated feedback3; similarly, whenever all aspects of user actions (and thus all user ac- tions) contained within a particular expression occur, the system invokes associated functionality. This principle is based on the associative nature of the RSL model, [Signer and Norrie, 2007a]. RSL had been successfully applied to PPI, in particular in combination with Pidget interaction, [Signer and Norrie, 2007b]. Furthermore, the concept of invocation in combination with semantics of expressions lays the founda- tion for machine understandable representations of interaction as discussed below in section 4.1.3.

3

In actual applications, feedback is of course optional. This can be conceptually modeled as ”invoking empty feedback”

Composition. Supporting composition of interaction techniques is a central re- quirement toward conceptual frameworks of PPI (R3.2). This entails two essential aspects of composition: on the one hand, conceptual frameworks need to support composing interaction techniques out of domain vocabulary; on the other hand, con- ceptual frameworks need to support composition of interaction techniques out of other interaction techniques. InW5

, expressions consists of composite chains of interaction predicates and other (sub-)expressions. As will be shown in section 4.1.3, composi-

tionthereby maps to logic operators connecting the interaction predicates. This en- ables composition at the syntactic level. However, composition at the semantic level of interaction predicates requires the additional concept of relative predicates.

Consider the examples of interaction predicates introduced on page 147. These predicates are absolute, i.e., they correspond to an absolute value observed in the digital ink produced by user actions. This concept becomes clear when looking at the example given above for the occurrence interaction predicate: here the system compares input with a pre-defined location (absolute value). However, specifying all input by means of absolute interaction predicates is problematic as they do not support conceptual relations of different aspects of user actions. However, this concept is needed toward enabling composition of expressions out of (sub-)expressions.

W5

addresses this problem by introducing relative interaction predicates. As ab- solute interaction predicates, relative interaction predicates coincide with one of the framework’s dimensions, e.g., one of the five core dimensions above. However, in contrast to absolute interaction predicates, relative interaction predicates describe re- lations of aspects of user interactions. An example for a relative spatial interaction predicate is ”above” meaning that one (aspect of a) user action must be performed above another, e.g., in the 2-dimensional pattern space. Another example for a rel- ative interaction predicate is ”after”, stemming from the temporal dimension. It de- scribes a relation where two (aspects of) user actions must be performed in a temporal sequence. For example, one can combine two spatial interaction predicates with the temporal sequence predicate (one shortly after the other) to describe, e.g., the double- click technique known from GUI systems.

Openness. Openness and extensibility at the conceptual level is another essen-

tial requirement toward conceptual frameworks of PPI (R3.3). Therefore,W5

was designed following the principle of openness: completeness of dimensions or pred- icates is not required in the framework and cannot be assumed at any point. W5

instead defines the structural constituents of the framework, i.e., expressions, interac- tion predicates and dimensions, as well as a relevant set of core dimensions and core predicates as part of the framework. Toolkits based on the framework need to reflect this by basing on the structural constituents, as well as offering core dimensions and

core interaction predicates. Thereby, core dimensions essentially provide a relevant conceptual frame for analysis of aspects of elementary user actions (and potential structuring to toolkits based onW5

); while core interaction predicates provide rele- vant aspects of user actions, i.e., aspects that are actually used in existing interaction techniques.

Extensibility is then achieved by allowing additional dimensions and interaction predicates to be added to the toolkit. For instance, a toolkit based onW5

needs to offer all the empirically derived core interaction predicates constituting the framework (c.f., section 4.2). However, it has to be designed extensible so that developers can easily add additional interaction predicates as required by their applications.