Integrating New Input Modalities into Interactive Applications

By adopting McMillan’s (McMillan 1995) paradigm as the high level conceptual model of interaction for systems which incorporate sensing, it is possible to begin to think about how best to provide support for the development of this type of interactive system. The main focus for consideration has to be the pre-processing of input signals that takes place in the sense-interpret- command portion of the model shown in Figure 4.12.

The first level of signal pre-processing involves detecting raw data from a sensor and converting that data into a format useable by an application. This may simply involve reading digital signals for the device port and converting them into floating point values for example. Some applications will be more demanding than others in terms of what is a useable format for the data, therefore a second level of pre-processing may be required. For example, applications that depend upon gesture recognition require a controller25 _{to deduce the user’s intentions from the continuous}

stream of input information. This deduction requires support in the form of learning algorithms or neural networks, which subsequent to a period of training perform real-time pattern matching on the incoming data stream. Once the user’s intention has been interpreted, a command can be issued to that part of the system best geared to deal with the command, be it at the interface or the underlying application.

To support the development of sensing-based interactive systems, a method of providing signal sensing and first level signal pre-processing is required. This must be provided in a manner that allows subsequent layers of processing to be added as required by a particular application. Furthermore, the signal processing must exist apart from the interface, for as shown in Figure 4.12 the results may be commands to either interface level entities or application level entities. Therefore, systems based on sensing require signal pre-processing to be carried out apart from both the application and the interface.

4.6.2 Discussion

By reviewing the types of tools that have emerged to support the development of interactive systems, it has been possible to identify a number of features common amongst them. The first is that tools for developing a certain class of systems adhere to high level conceptual models of interaction. So tools for developing WIMP-style interactive systems support the Seehiem, MVC or PAC model of interaction. Similarly, VR toolkits are designed to support the development of a combination of the environment, world model and application.

The input event streams generated by sensing devices are non-discrete and time varying. Continuous input devices are most often supplemental to existing discrete input devices such as mice and keyboards and the interleaved event stream (and multiple dialogue controllers) soon becomes extremely complex. It is not appropriate to try to force an existing interface-building tool to support the signal processing and presentation requirements of continuous input devices. In the first instance, continuous data streams should be dealt with separately from discrete user events, as they are subject to different degrees of pre-processing before they are suitable for presentation to either interface or application level entities. In the second instance the presentation mechanism for a continuous input device may be a continuous output device not necessarily of a nature found in traditional interfaces.

McMillan’s paradigm therefore provides a conceptual model which can be used to inform the design of tools to support continuous interaction events generated through sensing devices such as those found in EPIC systems. The model, like those describing WIMP and VR interaction includes both clearly identifiable high level components as well as a well-defined architecture.

The three components labelled sense, intepret and command serve to define the sensing model of

interaction from previous models26_.

Within this work we have chosen as our application domain systems that incorporate physiological information. These so-called EPICS provide specialized sensing input devices that produce continuous data streams that require pre-processing. Rather than try to extend an existing toolkit to include pre-processing functionality as well as support a suitable sensing device, we have chosen to implement both the signal detection and pre-processing functionality within a suite of reusable software components adhering to the JavaBean standard. The following chapter provides a description of the design, implementation and application of these Java components

4.7 Summary

This chapter has provided a review of tools to support the development of interactive systems. It was argued that the existence of high level conceptual models such as Seehiem and MVC have served to influence the design of tools which can be used to support the development of any system whose architecture reflects these conceptual models. Tools aim to support the development of one or more of the identified components in a high level model, such as the interface or the application. High level models can be used to reason about the kinds of dialogue that may exist between a user and an application. In the case of WIMP systems, this dialogue is driven by discrete user interactions with recognisable interaction techniques provided by the interface facilitated through the use of discrete input devices.

An alternative high level model of interaction was also presented which reflects the architecture of VR systems. This model partitions VR systems into 3 separate components, the environment, the world model and the application. Toolkits designed to support the development of VR systems were seen to provide tools for developing one or more of these components as well as the communication that would take place between them. Where WIMP interaction was driven by discrete user-generated event, VR interaction was seen to support a multi-modal approach to interaction, relying in part on continuous user-input devices such as motion trackers.

A conceptual model of interaction was identified for systems that incorporate continuous input devices used for sensing the user’s actions. A key component of this model is involved with detection and pre-processing of signals in the input stream. In order to support the development of sensing-based applications, and in particular those that incorporate electrophysiological information, a new approach was proposed based on the incremental construction of high-level software components. These components support sensing, first level pre-processing and then the addition of subsequent levels of signal processing based on the requirements of the application. The details of these components are discussed in the following chapter.

Chapter 5