Letrasprovides an extensible platform for rapid development of (mobile) PPI based applications. It empowers application developers to base on a common infrastructure and processing model. Letras is designed to support all pipeline stages, from acquir- ing digital pen data at the hardware interface (driver stage) to higher-level semantic processing needed by specific applications (application stage), supporting lightweight protocols and communication mechanisms. Letras also offers an application model based on interactive regions and associated functionality, that supports both local and remote applications.
Thereby, Letras employs a gray-box framework approach: application developers can construct instances of the mPPI processing pipeline using a set of predefined com- ponents which can ”plug together”; or developers can choose to extend and customize components if required by the targeted setting. In addition, highly flexible deploy- ment schemes as described in section 3.2.1, allow for easy adaption to the specific requirements of the environment at hand.
A ready to use, platform independent implementation of Letras is available for download10. The implementation supports the development of mobile PPI based ap-
plications on Microsoft Windows, Mac OS X and Linux operating systems. Due to the flexibility and small footprint of the underlying ubiquitous computing middleware, it is also possible for Letras to integrate applications and processing stages on resource constraint devices, such as smartphones, tablets etc.
Required adaptions of the base architecture for the mobile platform use case are described en detail in section 3.3.2. As reference implementation of MobiLetras, the mobile prototype of the Letras platform has been developed for the Android11operat- ing system for smartphones (version 2.1 and above). It has been deployed and tested on a Motorola Milestone mobile phone as well as Samsung Galaxy S and Nexus S smart- phones.
10https://github.com/fheinrichs/letras (accessed: July 2015)
Underlying Middleware
Implementing a plug and play approach toward flexible connection of pipeline stages, as laid out in section 3.2.1, is ideally based on a flexible and lightweight middleware. This reduces the amount of custom development for problems not specific to mPPI infrastructures, e.g., network transparent communication and service discovery. Such a middleware needs to provide suitable communication abstractions and supporting network transparent coupling of processing stages. Several such approaches exist in the domain of pervasive computing, e.g., Gaia [Rom´an et al., 2002] or MundoCore [Aitenbichler et al., 2007]12. As such, the middleware is not part of the actual mPPI infrastructure, it rather exists as an independent foundation.
Multiple ready-made middleware solutions exist. The criteria a middleware needs to support in order to serve as foundation for the mobile PPI infrastructure based on the distributed processing pipeline are
• network transparent communication both Remote Method Call (RMC) based to construct client / server links, as well as Publish/Subscribe (pub / sub)
• topic based pub / sub communication using unique, named communication channels (topics)
• services and a service discovery mechanism allowing for construction of service
oriented architectures(SOA) and micro service architectures (MSA)
Based on such a middleware, applications in Letras can then either detect deployed processing stages in their environment using service discovery, or provide their own stages. Connections between processing stages, as defined by the respective process- ing stage interface, rely on a completely transparent communication link: it does not matter, whether the connection is local, or on a remote device.
In order to realize this, Letras is constructed on top of the ubiquitous computing middleware MundoCore, [Aitenbichler et al., 2007]. It offers topic based publish /
subscribe(pub / sub) and remote method call (RMC) communication. In addition,
MundoCoreoffers a highly flexible component model based on services: each logical node in the MundoCore overlay network can host a number of services, while other nodes can discover these services, either deployed on the same logical node, or on any other node in the currently configured overlay [Aitenbichler et al., 2007]. Letras bases on this highly flexible, service oriented component model of MundoCore. Following this approach top-level components (processing stages) can simply be (re-)used or replaced according to the needs of the application at hand.
MundoCoresupports language independent communication channels between log- ical nodes in the Mundo overlay network. Ports for multiple programming languages exist in MundoCore, making it available for broad variety of platforms. This flexi- bility holds for Letras also. It is easily possible, for example, to develop a C# based application, which uses a Java based processing pipeline. Letras itself is developed in the Java programming language, although, due to the flexibility of the underlying
MundoCoremiddleware, any stage in the pipeline can be replaced by an appropriate stage in any other programming language for which a MundoCore port exists13.
Deployment Schemes
The underlying MundoCore middleware offers the concept of nodes, each of which hosts a set of services communicating via pub / sub channels. Effectively, even the provided RMC functionality is emulated over pub / sub channels. Service discovery is available, both on the same node, i.e., the same physical machine, or within the overlay network of Mundo nodes. Communication is handled network transparent, i.e., the middleware enables nodes to communicate either over a network transport or locally without exposing any additional complexity to the PPI infrastructure. In the local case, special optimized communication is used. A more detailed overview of the
Mundoarchitecture can be found in [Aitenbichler et al., 2007]. For details regarding concepts and implementation please refer to [Aitenbichler, 2006].
Plug and play interoperability as described in section 3.2.1 to realize different de- ployment schemes uses the MundoCore continuous service discovery function to im- plement the hotplug interoperability between processing stages. Here, services wrap resources, e.g., digital pens, and offer communication channels to receive data gen- erated by these resources, e.g., digital ink. Communication channels are realized via
MundoCore channels, allowing for topic based pub / sub. Additionally, the Mundo-
Corepub / sub channels allow multiple fan-out and multiple fan-in via subscription to a certain channel. This makes a scalable and flexible composition of interaction processing pipelines possible.
Each processing stage is wrapped in special Mundo service that starts all required services and monitoring processes of that stage. As such, it is only possible to run one instance of each stage per node. Therefore, the deployment scheme is determined by which Mundo nodes are started in the local network and by which processing stages are configured to be run on each node. MundoCore nodes can be configured to run certain services in a specific XML configuration file, the node.conf.xml. Hence, deployment schemes are determined by settings in the configuration files of the nodes available in the Mundo overlay network.
Distributed mPPI Processing Pipeline
As described above, each processing stage consists of a wrapping service, that starts or stops services for associated resources such as digital pens, interactive regions, seman- tic processors and data stores or rendering components. These services are themselves not discoverable, only the resource wrapping services they expose are discoverable by other pipeline stages.
Driver Stage The driver stage wraps MundoCore plugins containing drivers for
different pen models. Each driver handles a type of digital pen and will spawn a new
PenService wrapping each digital pen that is discovered. Drivers can be deployed in a hotswap fashion, that is, bundled plugins for new drivers can be installed into a running driver stage. Drivers could be wrappers accessing OS drivers or native drivers, i.e., drivers developed directly within Letras. As an example, Letras supports three native pen drivers:
• the Nokia SU-1B
• the Logitech IO2 (Bluetooth) • and the Anoto ADP-301 A
The Nokia and Logitech drivers are cross platform drivers developed in pure Java where communication with these pen models is handled via bluetooth14. This is possible, as both pen models stream their data using the bluetooth Serial Port Profile (SPP)15 on dedicated bluetooth channels as soon as the pen has been paired with a
device. They differ only in the channels they stream on and in the reference coordinate system of the digital ink provided, so their drivers have to provide an appropriate abstraction here.
The ADP-301 pen, however, uses the Human Interface Device bluetooth profile (HID), and requires a handshake to setup the pen connection before starting to stream its data. This involves operating system specific code as most operating systems in- tercept HID device connections, e.g., to support wireless keyboards and mice. As a result, part of the driver is C++ based and thus different ports for different operating systems exist16.
14based on Bluecove, a JSR-82 wrapper library for various native bluetooth stacks (http://bluecove.org/)
15https://www.bluetooth.org/en-us/specification/adopted-specifications (accessed: July 2015)
Region Stage The implementation of the region stage follows the principles out- lined in section 3.2.2. It uses a MundoCore continuous service discovery to discover all pen services independent on which driver stage they are hosted. At the same time, it uses a continuous service discovery to detect available interactive region services (i.e., services advertising IR descriptions) in the local Mundo overlay and constructs the bounding volume based LIRM out of these. Whenever digital ink is received on any of the channels of discovered pen services, this digital ink is dispatched to the channels of interactive regions containing it, if any.
Implementation is available in pure Java, as well as in the Android subset of Java. Here, it was important to not base geometry computation on the AWT17 classes, as
these are not existing on the Android platform. Hence, geometry functionality for rectangles, containment testing etc. had to be implemented within Letras itself to avoid different branches of code for the mobile / desktop versions within the pipeline itself.
Semantic and Application Stage Both semantic and application stage are also
written in pure Java. The semantic stage of the reference implementation includes a configurable segmentation service based on simple geometric as well as temporal proximity, a generic interaction processing service to interpret interaction techniques based on the conceptual framework of interaction introduced in chapter 4 allowing to quickly implement different interaction techniques, as well as several gesture recog- nition services, e.g., the 1$ gesture recognizer [Wobbrock et al., 2007] and a custom geometric recognizer for pie based gestures.
The application stage of the reference implementation stores the digital ink tree as produced by the region stage and the semantic stage (e.g., through clustering). It of- fers a persistence service based on MongoDB, a no-sql database18, to store the digital
ink data tree. On Android devices, a SQLite based implementation is used instead. Additionally, it includes two different rendering services: a Java Swing based imple- mentation for desktop applications and an Android version for use with the mobile platform based on Catmull-Rom spline interpolation [Catmull and Rom, 1974].
Interactive Region Publishing and Discovery
As described in sections 3.1.3, 3.2.3 and 3.2.2, Letras uses the two level interactive region and discovery approach. This is implemented using the MundoCore service discovery functionality within the local Mundo overlay network in the region stage of the pipeline. Currently, the reference implementation of Letras supports only local
17http://docs.oracle.com/javase/7/docs/technotes/guides/awt/index.html (accessed: July 2015)
Figure 3.12: Mobile platform support for the distributed PPI processing pipeline
region publishing and discovery to the full extent. However, this allows deploying and testing all PPI based applications on a local scale for real-world test cases.
Although the IRNS has been conceptualized as a CAN based DHT (c.f., section 3.1.3), Letras skips actual implementation of this concept as the underlying Mundo-
Coremiddleware offers a ready-made P2P service discovery functionality: a Tapestry [Zhao et al., 2001] based DHT. This mechanism can be used to discover remote region stage instances (and thus their LIRMs), which maintain non-local parts of the global region model.