Presentation and device matching

This section describes the interaction sequence steps 8 to 11. After the UIAF system generates the complete schedule table, it then computes all feasible presentation and device matches. The comparison takes base media items and matches them to the available devices. A content comparator mechanism is introduced in order to facilitate this, related to the interaction sequence step 8. The comparator mechanism allows the comparison of the media item and the device properties. It returns a matching quality confidence coefficient. In order for this comparator mechanism to work content and device properties need to be comparable, as has been assured by the design of the device and modality description format in Section 4.1.2.

During the parsing phase described earlier, the presentation schedule is generated. Each of the media items properties have been processed and are ready for comparison. The input for the matching step is the table of based media items including alternatives. Further, available devices with their respective descriptions are fetched from the Device Handler component. Therefore, processed device information is available. For each media item and device, media comparators perform a match as described in the next sub-sections.

4.2.5.1 Media comparator and matching quality coefficient (MC)

Figure 28a shows the general idea behind the comparator mechanism. A media comparator gets a device description and a media item description as input. Then it performs comparison based on its internal definition. The result is a matching quality coefficient (MC), which on a scale from 0 to 1 provides a measure of how well the media item matches the device (0 = lowest match, 1 = highest match).

a) Media

Device 1 Media item

description 1 description

Matching quaiity coefficient (MC)

b)

[ m edia comp.

D evice pro p erty 1 Item pro p erty 1

D evice pro p erty 2 Item pro p erty 2

I D evice pro p erty 3 Item pro p erty 3

Figure 28: Media comparator mechanism a) Overview b) Property level matching

Figure 28b shows the media comparator’s internal behaviour. Each media comparator inherits the knowledge of how to weight media item and device property matching. The compare function is formalized in equation (1):

1 n

M C (dd,m id) = —^ d d .p r o p e r t) i ® mid.propertyt (i)

This equation defines the matching quality coefficient (MC) by the average values of compared device description {dd) and media item description {mid) property matches. In this general approach the comparison of the parameters is defined by the ® operator and therefore can vary for different implementations. For example, the operator could be “=” or a defined value range. Equality is important when comparing the device’s general modality type (for example, a video- capable device can’t necessarily display a text or HTML media item). Matching quality coefficients for all device and content matches are recorded within the UIAF system’s internal data tables. Table 7 shows a theoretical example of a media items and device matching table with the calculated MC based on the previous introduced presentation schedule.

Base items

Media items for the matching Devices MC

Item 1 (text)

Item 1 (text) Device 1 (mobile) 0.6

Item 1 (text) Device 3 (TV set) 0.8

Alternative item 1 (text-to-speech) Device 1 (mobile) 0.8 Alternative item 1 (text-to-speech) Device 2 (HI-FI) 0.4 Item 6

(image)

Item 6 (high-resolution image) Device 3 (TV set) 0.9

Alternative item 6 (low-resolution image) Device 1 (mobile) 0.8 Item 8

(video)

Item 8 (high-resolution video) Device 3 (TV set) 0.7

Alternative item 8 (low-resolution video) Device 1 (mobile) 0.5

Table 7: Example media items and device matching table

Every feasible media item and device match shows the assigned MC. Alternative media items have been provided by the Presentation Adaptation component. Media items in the table are grouped by base media items, in order to highlight their relations. Media items in this example are text, picture and video with different quality values. Devices are a mobile phone, a TV set and a HI-FI set. Each of them provides different characteristics as stored in the device properties.

In step 9 of the interaction sequence the feasible presentation-device combinations are derived with the help of the matching table. From the example in Table 7, this step calculates two combinations as shown in Table 8.

Item 1 (text) MC Item 6 (picture) MC Item 8 (video) MC CC

1 Device 2 (HI-FI) 0.4 Device 3 (TV set) 0.9 Device 1 (mobile) 0.5 0.6 2 Device 2 (HI-FI) 0.4 Device 1 (mobile) 0.8 Device 3 (TV set) 0.7 0.63

Table 8: Feasible presentation-device combinations (MC = matching quality coefficient; CC = combination quality coefficient)

The combination quality coefficient (CC) is derived by calculating the average score of each matching quality coefficient (MC) for each possible presentation combination. The results are scored presentation-device matching’s. In example table the combination 1 has a coefficient of 0.6 and the combination 2 has a coefficient of 0.63. At this stage of the decision process the second combination would provide a slight advantage over the first combination in terms of matching quality, when applying the presentation to the current available devices in the user’s environment. In this example it is assumed that one media item should be assigned to one device. Therefore one device provides one Device Agent. In more complex scenarios devices can provide multiple Device Agents. This is planned for future work. Using the CC the system produces all feasible presentation-device matches. At that point, the system will include the user’s personal preferences, taking the user’s current context into account. Therefore the next section describes how the context-awareness fine-tuning is achieved.

4.2.5.2 Context-awareness fine-tuning

With the help of the combination quality coefficients the basis for step 10 of the interaction sequence is provided. Personalisation of the decision is achieved by applying the user’s personal preferences to the feasible presentation and device matches. This is named context-awareness fine-tuning. In this approach, user preferences relate to learned rules about the media item, device, and context configurations (e.g. the context can be location, user activity, etc.).

An approach linked to the first version of the UIAF system has been described in [78]. Herein the personal context function (PCF) and the personal profile function (PPF) are part of a personahsation system, which allows the context depended adaptation of multimodal applications, as envisioned in this work. Context parameters are collected and interpreted in order to create user preferences. Later the preferences can be applied by the context inference step, as described in the publication, in order to derive a context matching. Preferences in such context-aware systems are typically pre-defined or could be learned. Learning is a very flexible approach in order to react to user behaviour and the changes occurring over time.

The UIAF system supports the process of learning user preferences. For this it can be connected to personalisation systems, which are providing learning functionalities. An association rule learning approach as described in [96] was investigated. Figure 29 provides an illustration how the UIAF communicates with the personalisation system and its internal components.

Personalisation system Presentation-device matching (1) UIAF Recommender J Recommendations (3)

1

Final decision snap shot (4)

Snapshot data

Figure 29: UIAF system personalisation

During the decision process the UIAF requests a recommendation providing the current presentation and device combinations (1). The recommender applies the learned rules stored in the personalisation system to the received request, incorporating current context knowledge (2). However, rules are produced by a learning system. By applying the different rules, the recommender provides a situation confidence coefficient (SC) in (3), in addition to the

combination quality coefficient (CC) introduced earlier. Using a weighting function as shown in equation (2) the final confidence coefficient (FC) is calculated:

FC = w,CC + W2SC (2)

The weights can be altered between better presentation quality or user situation adaptation, but the sum of Wj and Wj has a maximum of 1. Table 9 shows the final presentation-device combinations (FC), applying the calculation presented in equation 2 the results of the example are presented in Table 8.

Content 1 (text) Content 6 (picture) Content 8 (video) CC SC FC 1 Device 2 (HI-FI) Device 3 (TV set) Device 1 (mobile) 0.6 0.3 0.45 2 Device 2 (HI-FI) Device 1 (mobile) Device 3 (TV set) 0.63 0.7 0.665

Table 9: Final presentation-device combinations (FC) (CC = combined confidence coefficient; SC = situation confidence coefficient)

For the first row of Table 9, the SC is 0.3 and for the second 0.7. Using the previous calculated CC of both cases, the FC is slightly altered, producing a higher match for the current feasible presentation quality and situation.

This calculation result concludes the matching and binding score calculation. From here the system either can take an automatic decision, which in this instance is the second row, or the system could present the user with a list of the two choices on the portal device. The list presents the final presentation and device combinations, highlighting the second presentation according to step 11 of the interaction sequence overview.

After the final decision either taken by the system or the user, the UIAF sends a final decision snapshot to the situation learning system (4). The learning system then will add the snapshot to its internal database (5) and calculates new confidence rules available for the next request (6).

This prepares the UIAF for presentation delivery, when the actual media item combinations are delivered to the selected devices.