Semantics and Notation

Conceptual frameworks of PPI need to satisfy the requirement of providing machine understandable descriptions of interaction in order to form an eligible basis for PPI or mPPI toolkits (R3.4). Toward this end,W5

derives its semantics from logic pro- gramming. This is based on the observation that interpretation of user actions and the invocation based approach ofW5

correspond well to term solving problems com- monly addressed by logic programming approaches or production rule systems: given certain, observable inputDI, the system needs to decide for each possible interaction technique whether its associated functionality shall be invoked or not. Thereby, ex- pressions correspond to rules, while observed digital inkDI corresponds to facts in logic programming terminology.

Basics

As laid out above,W5

defines expressions as composite chains of interaction predi- cates describing elementary aspects of user actions. Thereby, expressions correspond to the input part of interaction techniques, while feedback and function are invoked upon observing input. Expressions are thereby formed in a subset of first order predi- cate logic. As such, their structural constituents are

Logical Constituents This encompasses conjunction (∧), disjunction (∨), nega-

tion(¬), as well as the two truth constants: true (T ) and false (F )

Algebraic Constituents This encompasses parentheses (”()”) and brackets (”[]”) identifying precedence in evaluating expressions, the equality relation and sym- bol (=) and an infinite set of variables denoted through lower case letters (in- dexed where need), e.g.,x, y, . . .

Symbols This encompasses interaction predicates with differing valence or arity as laid out below (depending on what they exactly express). Interaction pred- icates are denoted as upper case letters or symbols (which are not structural constituents of expressions); predicates with arity≥ 1 carry the variable(s) they apply to in parenthesis, e.g.,A(x), B(x, y), . . .4

This allows forming expressions. Note, that logical quantifiers (∃, ∀) are not re- quired in order to express interaction withW5

. However, they might be used while reasoning aboutW5

expressions. The same holds for further logical symbols such as

implication(→) or equivalence (↔). This enables implementing systems to optimize recognition of expressions, i.e., allows for more efficient solving approaches, without limiting the expressive power.

Given a set of observable input (facts), interaction predicates and hence compos- ite logic expressions either evaluate to true or evaluate to false. Thereby, facts are the actions carried out by the user with a digital pen as observed by the system (in combination with other relevant system input), here denoted asDI. Any interaction predicate, or expression which based on receivedDI evaluates to true is considered

invoked. Invoked expressions trigger the associated functionality of the interaction technique; similarly, invoked predicates trigger associated feedback.

Bindings. In order to allow for detecting invocation of expressions, W5

uses the concept of bindings which is commonly found in term solving approaches provided by logic programming languages [Lloyd, 1984] and relates to the logic concept of unification (see, e.g., [Baader and Snyder, 2001]). Bindings essentially describe an association of system input with an interaction predicate; that is, the bindingBA of

an interaction predicate A is the subset of DI for which the interaction predicate evaluates to true (if any) as expressed in equation 4.1:

BA = {x ∈ DI : A(x) = T } (4.1)

Detection of invocation of such a predicate then determines whether BA 6= ∅, in

which case it is considered invoked. Detection of invocation of a composite expression thereby corresponds to evaluation of the boolean expression with respect to received facts, where all contained predicates are evaluated and the composite binding is deter- mined. Thereby, the process of detecting invocation of expressions also encompasses unification of variables, i.e., bindings are returned in the variables and can be reused in the expression. Bindings for variables have to satisfy all predicates and logic state-

4

As shown in equation 4.5 on page 154, the hierarchical shorthand notation omits the variable in cases where it is not further used in order to increase legibility.

ments containing these variables. This concept allows relating multiple aspects of user actions (along heterogeneous dimensions) to a single action.

Consider for instance an expression consisting of two interaction predicatesA, B that apply to the same variable, i.e., that are required to be present in the same input: E = A(x) ∧ B(x). As determined through unification, the binding for the composite expressionE would be

BE = {x ∈ DI : (A(x) = T ) ∧ (B(x) = T )} (4.2)

As stated above, detection of invocation thereby depends on the returned binding for the composite expression. Equation 4.2 shows that this binding consists of those facts for which both, A and B are true. Using the semantic of invocation as defined above yields that the functionality associated with this expression is triggered upon receiving input, i.e., user actions, for which both aspectA and aspect B are present. This demonstrates how user actions with elementary, observable aspects in multiple dimensions can be expressed through multiple interaction predicates relating to the same variable.

Domain of Discourse: Interaction Predicates

System input, and in particular user actions expressed through interaction predicates represent the domain of discourse in W5

. As laid out above, interaction predicates thereby correspond to atomic, observable aspects of user actions along conceptual dimensions. Interaction predicates can be applied to any system inputDI and evalu- ated. Depending on presence of the atomic, observable aspect they refer to, interac- tion predicates then evaluate either to true (aspect is present inDI) or to false (aspect is not present in DI). Furthermore, relative interaction predicates express relations among other aspects of actions in order to enable semantic composition in addition to syntactic composition enabled by the underlying first order predicate logic.

Following the principle of openness, W5

defines no closed domain of discourse: it encompasses basic, empirically derived relevant predicates (c.f., section 4.2), yet remains open to extensions through additional predicates (and associated semantics). However, as argued above, W5

defines two semantically different classes of inter- action predicates: absolute interaction predicates and relative interaction predicates. Thereby, classes can be distinguished by their arity.

Absolute Predicates Each absolute interaction predicate has an arity≤ 1, i.e., is

a nullary or unary predicate. Nullary predicates thereby correspond to constants or external, non PPI input (c.f.,EXT predicate defined below). In contrast to this, unary predicates correspond to observable, atomic aspects directly relating

DI to absolute values or predefined conditions along the conceptual dimension of the respective predicates.

For instance, consider the spatial occurrence interaction predicate example dis- cussed in section 4.1.2 on page 147. It describes input occurring at a particular place, e.g., on an interactive region. As introduced below in section 4.2, this predicate is denoted as AtR(x) and evaluates to true, whenever it is applied

to DI lying inside the region R. Thereby, DI is related to the region R, an absolute value in the conceptual spatial dimension (W1). Here, the bindings

returned inx for AtR(x) would be all DI observed lying within R.

Relative Predicates Based on the concept of bindings, relative predicates relate (aspects of) user actions to other (aspects of) user actions. Relative interac- tion predicates thereby aren-ary predicates with n ≥ 2. Bindings returned by relative predicates correspond to those elements of DI for which the relative predicate evaluates to true respecting contained (bound) variables, e.g., for a binary relative interaction predicateP (x, y), the binding of its x variable is

BPx = {x ∈ DI : ∃y ∈ DI . P (x, y) = T } (4.3)

As exemplified in equation 4.3, relative predicates correspond to observable, atomic relations of two or more sub-sets of DI along the conceptual dimen- sion of the respective predicate. Relating two or more sub-sets ofDI provides the basis for semantic composition as it allows for expressions reflecting how observable aspects of user actions are connected.

For instance, consider the binary temporal sequence interaction predicate exam- ple discussed in section 4.1.2 on page 149. It describes two actions, or aspects of actions, occurring in temporal sequence, i.e., one after another. As intro- duced below in section 4.2, this predicate is denoted as y (x, y) and evaluates to true, whenever it is applied to two elements ofDI where x occurred before y in the conceptual temporal dimension (W2). As such this predicate relates two

facts in time and y (x, y) returns all bindings to x and y so that x occurred (or was observed) beforey.

W5

allows forming expressions consisting of composite chains of aspects of user actions through absolute and relative interaction predicates in combination with the mapping to logic programming.

Example. Consider the following example to illustrate how the concepts of abso-

lute and relative predicates, in combination with binding of variables, enables express- ing composite chains of user actions. Based on the example interaction predicates

discussed above, we aim to describe an (arguably nonsensical) interaction technique, where the user has to ”double click” on three icons on paper in succession in order to invoke some functionality. Thereby, ”clicking on icons” corresponds to observing DI at a place and can be expressed using the example AtRpredicate, while ”double

[. . . ]” and ”succession” refers to temporal sequences and can be expressed using the example y predicate (here used as both, binary and ternary relative predicate). In W5

, this interaction technique can then be expressed as

x = AtR1(x1) ∧ AtR1(x2) ∧ y (x1, x2) y = AtR2(y1) ∧ AtR2(y2) ∧ y (y1, y2) z = AtR3(z1) ∧ AtR3(z2) ∧ y (z1, z2) E E E = y (x, y, z) (4.4)

Thereby, equation 4.4 shows how the relative temporal interaction predicate y ex- presses a relation between the spatial AtRinteraction predicates. It also defines the

expressionEEE based on bindings to variables x, y and z, subsequently relating their re- spective sub-expressions in the temporal domain (”succession”). This demonstrates how expressions can be composed of sub-expressions by means of relative predicates inW5

.

Hierarchic Shorthand Notation. As can be seen in equation 4.4, expressing

complete interaction techniques can result in lengthy systems of equations introduc- ing a lot of variables simply in order to express relations among interaction predicates. While this is important for machine understandability, it does not aid legibility, e.g., for interaction designers conceptualizing interaction techniques. Therefore,W5

intro- duces the hierarchic shorthand notation. It allows substituting interaction predicates in-situ for variables iff variables are not reused anywhere else in the term and the se- mantics of the term are not changed. Using the hierarchic shorthand notation, equation 4.4 can be rewritten as

E

E_{E = y (y (At}R1, AtR1), y (AtR2, AtR2), y (AtR3, AtR3)) (4.5)

Note that the in-situ substitution of sub-expressions for variables is used recursively in equation 4.5. As such, this abbreviated notation emphasizes the hierarchic compo- sition of predicates and allows for considerable shortening of expressions.

The EXT Predicate Because W5

as described up to know only addresses PPI input, it is impossible to combine non-PPI input with PPI within the framework. How- ever, this is unrealistic: in real world systems other means to interact with the system

exist, e.g., mouse and keyboard in the desktop setting or multi-touch interaction with a smartphone in the mobile setting. To remedy this limitation,W5

introduces a special predicate calledEXT , that evaluates to true when any relevant non-PPI input occurs. Relevant here refers to ”relevant as part of the interaction technique”. This could, e.g., be pressing a button on a keyboard to switch between inking mode and command mode for the UI as in [Liao et al., 2008].