Distributed PPI Processing

The basic idea behind the distributed PPI processing pipeline is to decouple the pro- cessing stages in order to allow applications to share pipeline components and hence to provide shared access to resources. This setup allows for a distributed deployment, as opposed to the monolithic deployment described in section 3.1.1.

To facilitate this, processing stages are conceptually and functionally decoupled and successive processing stages are connected exclusively via so called processing stage

interfaces(PSI). Processing stage interfaces are clearly defined interfaces describing data format and available services which interconnect processing stages as depicted in Fig. 3.2.

Driver Stage PSI IPen Region PS Semantic PS PSI IRegion Application PS PSI IInk Application (A1) Application (A2) Driver Stage Application PS

Figure 3.2: The distributed PPI Processing Pipeline: Successive processing stages (PS) transform digital ink data deployed in distributed scheme to allow for sharing of resources between applications (example setup)

Each processing stage consumes data in a form defined by its input PSI and pro-

ducesdata in a form defined by its output PSI. It also offers a set of services allowing to obtain this data. Thus, processing stage interfaces define stages of data, commu- nication channels used and services offered; whereas processing stages generate data and offer defined services.

In the distributed processing pipeline, processing stages can be deployed indepen- dently and thus physically distributed within the runtime environment. This adds the deployment flexibility required in order to lay the foundation for mobile PPI process- ing support while at the same time supporting non-mobile use cases (as the generic processing pipeline becomes one of the possible deployment options). A gray box around each processing stage indicates this concept of decoupling and physical distri- bution in Fig. 3.2.

There are three processing stage interfaces defined in the distributed PPI processing pipeline. The name of each PSI indicates the main data construct or service playing a role in the processing of PPI at this stage. The processing stage interfaces are

IPen At this stage in the pipeline the main data construct is referred to as raw digital

ink, i.e., digital ink data that has not yet been further processed. It consists of samples and events. Thereby, samples describe the position of the pen tip, while events indicate interaction events, e.g., putting the pen to the surface. However, the most important abstraction at this interface is the generic pen service: a soft- ware service, or more precise a collection of services, that hides connection and pen model specific methods and provides a uniform resource access to all avail- able digital pens. Hence the name of this PSI. This PSI decouples all sources of pen data from the rest of the pipeline.

IRegion Here the raw digital ink has been related to its enclosing region(s) and been processed accordingly. The main data constructs therefore are region related

has been clustered into traces which describe movement sequences with the pen on the surface. This digital ink consists also of samples and events. These are now dispatched to components interested in the interactive region the interaction occurred on. This is achieved by offering services encapsulating the data on specific interactive regions only, thus giving this PSI its name. Here uniform access to all defined interactive regions and their data is provided.

IInk Here the digital ink on a certain interactive region has been semantically pro- cessed. This adds meaning and interpretation to the data, thus further classify- ing the data structures obtained in the previous stages. Now the digital ink has been transformed into its final data structure, the processed digital ink. In this PSI, a set of services offer classification results obtained by semantic process- ing components, basically injecting these results into the pipelined data stream. The obtained digital ink is now ready to be used at the application level. Although these PSIs introduce additional complexity into the pipeline design, they offer numerous advantages in the context of mobile PPI processing. Fig. 3.2 shows how the distributed pipeline design allows sharing pipeline stages, both between in- stances of the pipeline and applications. This provides for an intrinsic non-exclusive access to resources, e.g., pens and interactive regions can now be shared by applica- tions. Resources are no longer locked into a specific pipeline deployment. This lays the foundation for infrastructures supporting flexible deployment (R1.3) and resource sharing (R2.1).

Furthermore, the connections of two successive processing stages via PSIs can be realized using a networking middleware. This makes it possible to distribute the pro- cessing stages physically, depending on the available computing resources. Ultimately it enables flexible deployments (R1.3) and operation on the mobile platform (R1.2), as resource intensive processing tasks, e.g., as part of the semantic processing stage, can be delegated to backend servers (assuming existing connectivity etc.).

Essentially, connecting successive processing stages via PSIs over communication channels offered by a networking middleware further increases the flexibility of the concept of distribution. It also allows for a wide spectrum of so called deployment

schemes.

Deployment Schemes

Deployment schemes describe the deployment of the pipeline stages, their physical and or logical distribution and their interconnection at runtime. In the narrower sense, a deployment scheme describes where a processing stage will be executed and which processing stages will be shared by multiple instances of the pipeline. In the broader sense a deployment scheme describes the general processing paradigm employed by

PSI IPen PSI IRegion PSI IInk Application (C1) Application PS Region PS Driver Stage Driver Stage Semantic PS

PSI IPen PSI IRegion PSI IInk

Application (B1) Application PS Region PS Region PS Driver Stage Driver Stage Semantic PS Semantic PS

PSI IPen PSI IRegion PSI IInk

Application (A1) Application PS Application PS Region PS Region PS Driver Stage Driver Stage Semantic PS a. b. c.

Figure 3.3: Examples of deployment schemes in the distributed processing pipeline: a. note taking application with a desktop and mobile client, b. the shared whiteboard application with personal pens, c. the borrowed pen setting

the pipeline. The distributed processing pipeline enables a broad variety of different deployment schemes. This is a crucial advantage over a fixed monolithic deployment when it comes to support mobile usage practices as it allows the pipeline adapting to the current use case as defined by the application (or the set of applications).

In order to illustrate the flexibility of the concept of deployment schemes in the distributed processing pipeline, consider the three example schemes depicted in Fig. 3.3.

The first example corresponds to a note taking application allowing the user to employ PPI in order to edit notes. It features a desktop and a mobile client, both supporting input via digital pens. The mobile client delegates processing intensive tasks of the semantic processing stage, e.g., handwriting recognition, to the desktop pipeline in order to save battery. Its deployment scheme is depicted in Fig. 3.3 a., the mobile client being the top row.

The second example depicted in Fig. 3.3 b. is a shared whiteboard application where users can interact with their personal pens on a collaboratively used whiteboard. In this deployment scheme, the first processing stages are deployed on the individual mobile clients connecting with the whiteboard. Only the rendering and data storage, i.e., the application processing stage, is handled on the whiteboard server (indicated by a gray background of the application PS in 3.3 b.).

The third example is a mobile PPI based application supporting sharing of a pen between different users. Here, the cumbersome manual setup of the bluetooth pen connection is abstracted away by the flexible pipeline design. The driver stage remains deployed on the mobile device of the first user lending out her pen. All further stages are deployed on the mobile device of the user actually using the pen for interaction. Fig. 3.3 b. depicts this setting.

In this broad range of heterogeneous deployment schemes, the monolithic deploy- ment itself can be considered a degenerate case in the context of the distributed pro- cessing pipeline: Even with the distributed pipeline architecture, it is possible to de- ploy all processing stages alongside the application. This then yields the monolithic deployment found in most existing infrastructures (c.f., section 3.1.1).

Another deployment scheme emulating the approaches taken by existing infrastruc- tures is to host the driver stage on a client device, while the other stages are hosted at a central server. This then yields the client / server approach, e.g., as taken by iS-

erver / iPaper, [Norrie et al., 2006a]. Thus the flexibility of the distributed approach enables deployment as in existing architectures while at the same time adding support for alternate deployment schemes supporting the mobile use case.

Sharing of PS. Sharing of processing stages (PS) between applications is thereby

facilitated through two concepts: services and interfaces, as described above, and the

pipeline base architecture. While services and interfaces ensure that decoupling and physical distribution is possible, the pipeline base architecture, where data continu- ously moves forward through the processing pipeline, ensures it is feasible. Services thereby are not required to be completely state-less, e.g., recognition services will track and change their state according to the samples and events received through the pipeline. Re- entrant usage of services as such, however, does not occur as data successively travels through the pipeline.

It is important to note here, that back channeling of data has to be possible: Con- sider, for instance the semantic stage as recognizing a certain gesture, that would then trigger handwriting recognition of previously recorded digital ink. In this case, in- frastructures based on the paradigm of the distributed PPI processing pipeline need provide adequate means to enable such concepts, e.g., through a flexible Micro Ser-

viceapproach in the semantic and applications stages (c.f., section 3.2). Required and Optional Stages

Not all processing stages are essential to all applications. Consider for example a simple drawing application for multiple users, which does not require any semantic processing of PPI. Also the rendering is done by the application itself and it stores only a bitmap version of the resulting drawing. Such an application requires only the Driver

Driver PS PSI IPen Region PS Semantic PS PSI IRegion Application PS PSI IInk

Required Stages Optional Stages

Figure 3.4: The distributed PPI Processing Pipeline: required and optional processing stages

PS to connect the pen hardware and the Region PS to identify whether the interaction did occur on the drawing region. In the context of the distributed processing pipeline, this concept can be used to further classify available processing stages

As processing of digital ink at the very basis means connecting a digital pen as in- put source and relating ink to an interactive region, e.g., a paper document, the Driver PS and the Region PS form the required processing stages, whereas the Semantic PS and the Application PS form the optional processing stages3_{. As trivial as this}

distinction might sound at first, it is important: focusing on the required stages and optimizing their implementation yields better resource utilization. This enables the pipeline to adapt more flexibly to mobile settings and to the needs of applications. Thus, it reduces the deployment size and provides a lower footprint of required soft- ware components. Ultimately, this supports operation on mobile platforms better.