Device and modality description

Mobile context-aware user interface realisations, such as the UIAF, demand for a complete and accurate device and modality description. They provide the means for enabling any kind of matching and decision making in any occurring situation for both user interaction interpretation as well as multimedia delivery decisions. Existing device descriptions are not necessarily designed to encompass Mobile Multimodal User Interfaces. Differences between available discovery mechanisms do not allow discovering a rich and uniform set of information about devices in the vicinity. For example, whereas the Bluetooth protocol allows discovering devices and their Bluetooth profile services, the Wireless LAN discovery stops at the device level.

Three main factors are important when addressing device descriptions in Mobile Multimodal User Interfaces. First the physical parameters of a device, second what modality capabilities it has to offer towards a multimodal user interface system and third how delivery or reception of modality information is supported by the network access of the device. Figure 22 shows this with an example of a mobile terminal or also PDA, which provides the most complex device class in terms of capabilities at current. Other examples can provide less or are more focused on a specific feature, as for example a display providing high resolution images and video, or a microphone for just providing high quality audio input.

Physical modalities - display (180x240, 256 colors) - speaker (1way) - microphone (8kHz) - pen input Network access - Bluetooth - wireless LAN GSM Modality services - video (display) - audio (speaker) - text (display) - speech (microphone)

Figure 22: Modality device example

Furthermore relationships between the three categories and the hosting device can be established. In order to capture the model, a description framework to describe uniformly the discovered devices and the modality services that they support is provided. This work, detailed in [93], relies on the principles shown in Figure 23 using UML notation.

HostDevice PhysicalModality _{-id .}

-id 1 1..* -id 1..* 1 -name

-name -name -URI

-deviceType -physicalDeviceType -modalityType

-capabilities -capabilities -modalityFormats

-capabilities 1..* 1..’ Networklnterface 4d -name -networklnterfaceType -capabilities

Figure 23: Device description definition - main categories and relationships

According to the illustration a host-device (HostDevice) is defined as self-contained piece of hardware, the actual device, with which the user interacts (e.g. the user’s mobile terminal, displays and various user interface devices). They are composed of a set of physical modalities

(PhysicaMmodality), which are the different pieces of hardware that render a given modality. For

example, in simplified terms a laptop is a host device, composed of a screen, speakers, keyboard and microphone. Furthermore, it is considered that these physical modalities are used through software components called Device Agents (DeviceAgent). Finally, these software agents are

accessed using one of the available network interfaces (Networklnterface) that are supported by the host device.

From this model, the UIAF business logic at first step is applying the device agent’s descriptions. Typically, information on Device Agents include their modality class (audio, video, text, etc.), the means to access it, and the supported data format as well as optional capabilities information. In case of an actual decision taken in the Modality Fission and Modality Fusion components of the DeaMon, the physical capabilities, and network capabilities are used in a second instance. The detailed matching mechanism is described in Section 4.2.5.

As most devices cannot provide a comprehensive set of information on themselves and the Device Agents they support, the description might be retrieved from a distributed database, as it is done in the UAProf [13] approach.

4.2 Multimedia presentation delivery

Being a substantial part of multimodal user interaction, multimedia presentation delivery often referred to as fission is one of the major research topics of this thesis. Particular to this work is the proposed adaptation approach and its suitability for multi-device scenarios taking into account current user context and learned user preferences. For instance, a user is driving a car. In order not to distract the user unnecessary from driving, the system should select the best suitable delivery channel in this situation. If a user receives a text message, the message should not be visualised on the mobile terminal, but rather played in audio format through the car HI-FI system. This can be achieved by applying a text-to-speech transformation of the text and select it for playback.

During the research work a general approach to multimedia presentation delivery evolved. In the first instance the adaptation and delivery process has been investigated using single media items matching their properties to the available device characteristics. This approach has been published in [94]. Further considerations lead to the conclusion that a generalisation of the approach for intelligent multimedia presentation delivery in multi-device environments is desirable. It offers a higher degree of flexibility and potential for an immersive user experience. Such an approach has not been researched in this form before. This section presents the related research results. Some general remarks about the invocation of the delivery process are given in the following.

Two events for triggering the adaptation and delivery process are defined. These occur when an application sends a presentation delivery request to the UIAF or a context change is discovered. A context change occurs when devices configuration change and this is detected through the discovery framework (see Section 4.1). Device configuration changes either indicate location changes (e.g. a car device is detected), or devices being introduced to the scenario (e.g. a TV is

switched on). The system reacts to a presentation delivery request, by generating an internal presentation schedule. This is repeated for each request and once a presentation is executed the system keeps track of these executed multimedia sessions. In the case of a context change all sessions are re-evaluated and depending on the change, sessions are re-routed to the new device configuration. A session is continued at the position the event occurred. With this outlined, the rest of the section presents the multimedia presentation delivery interaction sequence overview and subsequently the details on the involved steps.

4.2.1 Interaction sequence overview

This section provides the overview of the multimedia presentation delivery interaction sequence, highlighting the several involved UIAF components and their functionality in the process. For this purpose the UIAF initial system architecture introduced in Section 3.4 is adapted to highlight the involved components and data models. This is shown in Figure 24 together with the sequence numbers for each process step.

B A N / P A r

D ev ic e and D evice tailored

F u n c tio n (I p re sen ta tio n description D ev ic e a n d G atew a; F u n c tio n (D eG an ) Session Control Device A gent C o n tr o l F u n c tio n a lity Device Handler D evice d escrip tio n s

in p u t F u n c tio n a lity iu tp u t F u n c tio n a lity

Presentation Transform ation

Adaptation description Content Handler P rese n tatio n m odel Modality Fission Internal d a ta ta b le s UIAF Stub Application o utp u t re q u e st Mobile & W eb applications “ M ultimedia ~ Personalisation - d ev ice re c o m m en d atio n p re sen ta tio n description - co n tex t c ap tu re - le a rn e d u s e r p re fe re n c e s

Figure 24: UIAF multimedia presentation delivery sequence and data models

The system flow starts from the Mobile or Web application on the lower end, sending the output specification including its multimedia presentation description, evolves through the DeaMon towards the rendering Device Agents. For flexible and optimal multimedia delivery decision, functional components such as the Content Handler, the Device Handler, the Presentation

Adaptation and the Personalisation system are involved. Considering the number of steps and the complexity at hand, an initial overview and description of the sequence is provided in Table 5.

Nr. Description of the step Resulting data

1 Retrieve application output specification and multimedia presentation description.

Application output specification and multimedia presentation description

2 Parse the service presentation description into a language- independent data model that includes the media item parameters. Register it to the internal Content Handler tables, as a new output request for specific user and application.

Presentation description tree model from multimedia

presentation with link to user and application.

3 Build presentation schedule for the presentation description tree.

Internal presentation schedule

4 Trigger the adaptation decision process either through the application request event or context change event.

Adaptation trigger event (application request, context change)

5 Fetch registered output specifications including the active presentation schedules for each user and application.

List of presentation schedules for the user

6 Find the maximum parallel sequences and take the first trailing media item as the base item for matching.

List of single media items, representing media item sequences

7 Add alternative media items to presentation schedule. Query the Presentation Adaptation component about possible transformations.

Alternative media items linked to their originals, (two dimensional list)

8 Associate available devices to media items by comparing media item parameters with device capabilities and by assigning a confidence value.

Presentation-device matching confidence table

9 Compute feasible presentation-devices combinations presentation and device matching).

Feasible combination of presentation-device matching tables

10 Fine-tune the rating based on context recommendations, user preferences, and presentation consistency.

Fine-tuned ordered device- presentation tables

11 Select the first device-presentation matching, create the content-device binding, and forward to the delivery mechanism.

Binding table

12 Produce single presentation document for each rendering device.

Content delivery descriptions

13 Perform content adaptation. Updated content delivery

descriptions 14 Rendering of the multimedia content and synchronization

between devices.

Table 5: Multimedia presentation delivery sequence description

Based on the introduced table subsequently the multimedia presentation delivery steps are explained in detail.