Development Support Tools for Rapid Prototyping of Visual Output

The frameworks MOCA [Roman et al., 2000] and Aura [Sousa and Garlan, 2002] support rapid prototyping of visual output for desktop based visualization devices. In the MOCA architecture, programming logic and representation are again separated [Roman et al., 2000]. A service advertises itself via XML document. A UI is then generated technology dependently, downloading the XML before. For instance, the abstract XML description can be distilled into an HTML page or it can be represented by a Java application.

The framework Aura implements an architecture which defines user tasks as first class entities that are described as abstract services [Sousa and Garlan, 2002]. Service suppliers also have an abstract description. Aura now tries to find service suppliers for the requested services of the task when resources change, e.g., because the user is moving to a different environment. The architecture is not restricted to a particular resource so that it is also applied for visual output.

The Real World Interfaces library facilitates the usage of physical objects for visual output [McCrickard et al., 2003]. It provides an API for hiding complicated details of lower level programming languages. In addition, it handles hardware issues, such as error detection or interface blocking. A physical object is virtually represented through a proxy, acting as interface to the device for the programmer. The RWI library is limited to X102 _{devices. Rapid exchange of visualizations is not possible due to the lack of a}

common interface for all device proxies.

The aim of UbiWise is similar to the one of this thesis: Support the exploration of user interfaces and interactions of devices in ubiquitous computing applications [Barton and Vijayaraghavan, 2003]. For this, UbiWise provides a virtual 3D environment for which ubiquitous computing devices can be modeled (cf. Figure 2.7). The environment itself can be designed with the framework and interaction rules for the devices can be defined. Hence, UbiWise lets the user integrate new technologies but not in the real world.

The INCA infrastructure allows for capturing, storing, transducing and again access- ing of media data [Truong and Abowd, 2004]. Its Accessor module provides a standard- ized interface to the data which every output component can request. However, INCA is initially restricted to media data. Output technologies themselves are not provided by the toolkit and must be developed from scratch.

d.tools allows for rapid prototyping of physical UIs with a focus on system design [Hartmann et al., 2006]. It provides a number of input devices that can be combined and also supports the creation of output components that incorporate a small LCD display.

Figure 2.7: The use of a digital camera is simulated by UbiWise in a virtual 3D environment [Barton and Vijayaraghavan, 2003].

Designers place physical controllers, sensors and output devices on form prototypes. The corresponding behavior is authored in a software workbench. d.tools does not support the actual development process and is also not open for a general set of devices.

[Mori et al., 2004] support more steps of the application design process. At first, tasks need to be modeled. From these, system tasks are derived. Then, UI components for fulfilling the system tasks are identified for every device class. The outcome of every step is an XML file from which concrete UIs are automatically generated.

The EIToolkit supports rapid prototyping of pervasive applications by providing an architecture for connecting various input and output technologies [Holleis, 2008]. Each technology is connected by a particular stub, which acts as driver that translates from technology-specific code to EIToolkit messages. Although not specialized on exchanging visual output for a given application, the framework can be used to connect different output technologies to an application, as long as stubs which understand the same messages exist.

Several frameworks apply the model-view-controller pattern [Reenskaug et al., 1996] to generate device-specific visual interfaces [Roman and Campbell, 2002, Berti et al., 2004, Stirbu, 2010]. Developers can create abstract models of the views with the help of the frameworks’ toolkits. The frameworks translate the models into concrete imple- mentations that are adapted for the particular devices. The frameworks do not support ubiquitous annotation visualization as they do not consider the connection of virtual and real world. In addition, the supported devices of the earlier frameworks are outdated.

2.2. Related Development Support Tools Reverse engineering tools like [Bandelloni et al., 2008] try to analyze an existing user interface for a given platform and derive a general task model. This can again be used to adapt the user interface to other platforms. The general idea behind this procedure is to find an abstract representation for a user interface and to derive concrete implementations. Our UbiVis framework will make use of this idea.

The work which conforms most to our work is done in the scope of the interactive workspaces project [Johanson et al., 2002]. It features the iRoom, a workspace envi- ronment which consists of several interconnected ubiquitous computing appliances like smartboards, remote interactable lights or handheld devices. ICrafter is the framework for dynamically generating UIs for these appliances [Ponnekanti et al., 2001]. The Inter- face Manager, a central instance, is delivering the UI based on the requesting appliance. A template for each appliance is generating code (e.g. HTML), which is rendered on the appliance.

iStuff is a toolkit supporting physical input and output components (cf. Figure 2.8) [Ballagas et al., 2003]. An application in iStuff generates an event which is put on an event heap. Events must be mapped to devices, which can be done with the help of a graphical tool. The devices generate output and input based on this mapping and the events they receive from the event heap. iStuff also introduces several supported gadgets, most of them self-built. An extended mobile version of iStuff [Ballagas et al., 2007] is used for fast prototyping of existing mobile phones which are augmented by external sensors (cf. Figure 2.9). The mapping of application events to phone+sensor events is done via background application, which is provided by iStuff mobile.

Figure 2.8: A set of self-made output components for iStuff [Ballagas et al., 2003]

The work of [Greenberg and Fitchett, 2001] is similar to the interactive workspaces project. They present phidgets, which is a short term for physical widgets. Like a widget, which hides implementation details for the programmer, phidgets package physical input and output devices. [Greenberg and Fitchett, 2001] let physical designers create the hardware device and make it available to software developers via an event-based API.

Figure 2.9: A mobile phone is augmented by sensor hardware [Ballagas et al., 2007].

2.2.5 Discussion

The visualization tools in Section 2.2.1 are commonly used in the scope of other frame- works for the actual visualization task. Accordingly, the Mobile AR Framework [Billing- hurst, 2014] will be used for creating the UbiLens visualization in the scope of our framework (cf. Section 6.1).

Rapid prototyping support for pervasive applications is often provided by choosing from a set of pre-defined functionality (cf. Section 2.2.2). So, the assistance of these frameworks is restricted to this limited functionality. However, our aim is to leave the developer the complete freedom to implement everything possible in a standard programming language.

In general, most of the presented development support tools cannot be used for rapid prototyping of ubiAV as they address a different domain. Though, the survey gives us insight about successful concepts for the design of a framework, such as the separation of concerns or the ability for developers to extend framework code.

Separating the input layer from the rest of the application is a promising approach in several development support tools (cf. Section 2.2.3). But as the example of Papier- Mˆach´e shows, the concepts cannot be exactly transferred to separating the output layer [Klemmer et al., 2004].

The tools presented in Section 2.2.4 support the rapid exchange of visualization interfaces for ubiquitous computing applications. However, most of them do not support ubiquitous post-desktop visualization devices. Additionally, the tools only provide a restricted set of functionalities. So, the scope of applications that can be implemented with these tools is limited. Often, the tools are restricted to a certain aspect so that the developed prototypes miss important factors for user evaluations. In this sense, [Park et al., 2009] simulate haptic user experience. UbiWise simulates interaction experience but lacks of haptics [Barton and Vijayaraghavan, 2003]. Development support is missing for d.tools [Hartmann et al., 2006]. EIToolkit lacks of a defined interface, which makes the visualization exchange highly dependent on the existence of stubs that understand

2.3. Conclusion