Pidget Interaction: PaperPoint and PaperProof

PaperPoint, [Signer and Norrie, 2007b], and PaperProof, [Weibel et al., 2008], em- ploy the iServer / iPaper infrastructure for PPI. As such they represent an important

5

To be precise, these are interaction predicate classes, as each of the nine interaction predicates can be used in parametric form (absolute predicates), or as n-ary definition (relative predicates).

class of systems basing interaction on the RSL, [Signer and Norrie, 2007a], concep- tual framework of interaction which focuses mainly on invocation of PPI techniques. As discussed in chapter 2, section 2.4.3, RSL introduces the concept of link invocation through different selectors, e.g., regions on paper documents. This provides a natural basis for Pidget based interaction.

PaperPoint. PaperPoint, [Signer and Norrie, 2007b], represents an early approach

toward PPI. It revolves around interaction through Pidgets, combined with free form drawing and annotating. In PaperPoint, users can control functionality offered by the

Microsoft PowerPointpresentation software through a set of Pidgets. Users can print out an overview version of the slides contained in a presentation. The system then enables users to point the digital pen on icons representing slide show functionality, e.g., move one slide forward, start the presentation or continue at a particular slide number. Additionally, the system enables users to directly annotate print outs of slides and show a facsimile of the annotations in the presentation.

Here, two exemplary representatives of the interaction techniques offered by Pa-

perPointwere chosen in order to facilitate analysis of interaction predicates required toward expressing these techniques inW5

. The first representative is the technique for starting a presentation at a particular slide by tipping on a printed ”show” button with the digital pen. Modeling this interaction techniques inW5

requires an interaction predicate describing occurrence in the spatial dimension, i.e., theAtR(x) predicate

defined below. The second representative of interaction techniques offered by Paper-

Point, enables users to draw or write on a particular slide. The system then switches to this slide and displays a facsimile of the digital ink recorded on this particular slide. This requires the aforementioned occurrence predicate, as well as an additional interaction predicate expressing reception of uninterpreted6_{contentin the content di-}

mension, i.e., theC(x) interaction predicate.

The required interaction predicates are thereby defined as

AtR(x) The AtR(x) predicate describes occurrence of digital ink in the spatial di-

mensionW1. Thereby, this absolute predicate is a parametric predicate relating

to occurrence of digital ink at the interactive region defined by parameter R. As such, this predicate evaluates to true for all digital ink inx that lies within regionR.

C_{(x) The C(x) predicate is an auxiliary absolute predicate describing reception of} uninterpreted content for further use in an interaction technique. It stems from

6_{Uninterpreted here means that the system does not evaluate the content per se, it merely records it and}

subsequently uses it in combination with other interaction predicates or triggers functionality based on content (e.g., displaying its facsimile)

the content dimensionW3 and characterizes the need for entering digital ink,

i.e., handwriting or drawing, as part of an interaction technique. As such it al- ways evaluates to true binding received digital ink without any additional con- ditions, e.g., when aiming to record digital ink for using it as facsimile.

Based on the defined predicates, the Pidget interaction technique (E1) and the slide

annotation interaction technique (E2) can then be expressed inW 5 as E1 E1 E1= AtS1(x) E2 E2 E2= AtS2(x) ∧ C(x) (4.6)

Note that the parameterR of the absolute AtR(x) predicate used in equation 4.6 re-

lates to two different interactive regions in this example (withS1reflecting the ”show”

icon andS2reflecting the slide printout).

PaperProof. PaperProof, [Weibel et al., 2008], builds on the concepts introduced

in PaperPoint and adds semantic interpretation of content, thus employing interac- tion predicates along the content dimension (W3). PaperProof enables users to per-

form proof editing of documents directly on a paper printout. This includes inserting, deleting, replacing, moving and annotating text. Thereby it combines the spatial oc- currence with gesture based interaction techniques (c.f., discussion of PapierCraft, [Liao et al., 2008]) in order to express more sophisticated, chained interaction tech- niques.

Here, the two most complex interaction techniques offered by PaperProof were chosen for analysis: annotation and move. Interestingly enough, the informal no- tation employed by Weibel et al. to describe the interaction techniques offered by

PaperProof resembles the structure of expressions used inW5

, [Weibel et al., 2008]. Although it lacks some details, e.g., with respect to location of gestures, it can be read- ily transcribed. Thereby, annotation of text elements requires first enclosing the text in brackets (gesture), consisting of an opening followed by a closing bracket, and then writing digital ink, i.e., the actual annotation. Similarly, moving requires enclosing the text (as for annotation) and then entering a special line gesture.

Expressing these techniques inW5

requires two additional interaction predicates in theW2andW3dimensions respectively

G_{S(x) The absolute GS(x) predicate describes a gesture in the content dimension} W3. G_S(x) is a parametric predicate relating to recognition of a particular

gesture (i.e., one constituent of the gesture vocabulary). Thereby, the particular gesture is defined by parameterS. As such, this predicate evaluates to true for all digital ink inx that constitutes the gesture symbol S.

Technique Formalization

annotation y (AtR(Gcs1∨ Gcs2∨ G<),

AtR(Gce1∨ Gce2∨ G>), DI)

move y (AtR(Gcs1∨ Gcs2∨ G<),

AtR(Gce1∨ Gce2∨ G>), AtR′(G_N))

Table 4.1: Interaction Techniques in PaperProof, [Weibel et al., 2008]

y (x1, x2, . . .) The relative y predicate expresses a temporal sequence of its con-

tained variables (W2). W5 defines it as n-ary predicate with n ≥ 2, that is,

it can be used to express a temporal sequence of an arbitrary number of con- stituents. As such, y (x1, . . . , xn) evaluates to true iff ∀xi, xj.1 ≤ i < j ≤ n

the digital inkxiwas received before the digital inkxj. As such it allows relat-

ing sequences of (elements of) user actions.

Based on these predicates in combination with the predicates defined above, the

annotationinteraction technique (E3) can then be expressed inW 5

as

E3

E3

E3 = At_R(x) ∧ (G_CS1(x) ∨ GCS2(x) ∨ G<(x))

∧ AtR(y) ∧ (GCE1(y) ∨ GCE2(y) ∨ G>(y))

∧ C(z)∧ y(x, y, z) (4.7)

HereGCSi, GCEi, G< and G> are different start and end delimiter gestures for

marking text as defined in [Weibel et al., 2008]. The interactive regionR used in the AtR predicate refers to the paper document, paragraph or sentence being marked.

Bindings forx and y then express that both start and end gestures need to be executed sequentially in the same region, followed by an annotation that can be given anywhere. The produced annotation itself is bound toz in C(z).

As can be seen in equation 4.7, the full formal notation of real world interaction techniques inW5

can produce lengthy expressions. Therefore, the following exam- ples employ the hierarchic shorthand notation defined on page 154. Table 4.1 presents the hierarchic shorthand notation of expressions for both interaction techniques dis- cussed, annotation and move.