ConcurTaskTree

CTT which stands for ConcurTaskTree is a notation that was first introduced by Paterno [108] and which provides support for design and analysis of complex task models in multi-user environments. With it a compound activity is represented as a task tree, where each tree node represents a task which allows composite tasks to be decomposed into subtasks. Various tem- poral operators are used for expressing the relations between the tasks in the tree. Each task is associated with a specific type, a category, attributes, and objects it is able to manipulate. Ad- ditionally, it provides a graphical syntax that allows easier interpretation of the logical structure of a task.

Fig. 2.9 shows an example of a task model, where a simple composite task consisting of four tasks (A, B, C, and D) is represented in a CTT notation. The tasks A, B, and C can be executed in any order, which is specified by the temporal relation order independency (_{| = |).} Additionally, task D can be performed only after all the other tasks are executed, which is specified by the relation enable (>>).

In that manner user behaviour and the interaction between different users can be expressed in a tree-like manner.

A |=| B |=| C >> D

Fig. 1. Task model specifying the schedule of a meeting

compound activities. For inferring the activity of a user from sensor data, we need additional information: a specification of how probable a certain execution sequence is. Next, we will look at a current approach to this problem.

3 Inferring Intentions

As outlined above, computing the user’s current activity from sensor data re- quires a task model that allows to make statements about the plausibility of sensor data given a specific activity. A system can then try to identify the user’s current task by selecting that task, whose action sequence is most plausible with respect to the observed sensor data.

Bayesian Filtering for identifying a user’s current task has been successfully used in several projects that aim at supporting user activities in classrooms, meeting rooms, and office environments [4,5,6]. Here, dynamic Bayesian networks (DBNs) are investigated increasingly for modeling a user’s activities [7,8].

In our own work, we look at using DBNs for inferring the current task and actions of a team of users. Given (noisy and intermittent) sensor readings of the team members’ positions in a meeting room, we are interested in inferring the team’s current objective – such as having a presentation delivered by a specific team member, a moderated brainstorming, a round table discussion, a break, or the end of the meeting.

The basic structure of the DBN we propose for modeling the activities of such a team is given in Fig. 2. In general, a DBN consists of a sequence of time slices, where each time slice describes the possible state of a system at a given time t. A time slice consists of a set of nodes that represent the system’s state variables at that time. State variables may be connected through directed causal links. A connection such as X→ Y means that the current value of Y depends on the current value of X. This dependency is described by a conditional probability table (CPT), such as

X = 0 X = 1 P (Y = 0|X) 0.9 0.3 P (Y = 1|X) 0.1 0.7

which in this example says that, in case X is 1, the value of Y will be 0 with a probability of 0.3 and it will be 1 with a probability of 0.7. (If X is 0, Y will be 1 with a probability of 0.1 and 0 with a probability of 0.9.)

Figure 2.9: CTT for a composite task with four tasks (A, B, C, D) (Figure adapted from [58]).

2.5.3.1 Applications

CTT is mostly used in human-computer interaction problems such as building successful interface designs. For example, Li at al. [89] use CTT to generate interface model of a display/- control system. Furthermore, Klug et al. [77] extend CTT to accommodate its execution during runtime, allowing the generation of applications that adapt to the user actions and preferences. It has also been shown that CTT can be applied in the field of activity recognition. In their work [58], Giersich et al. use CTT to model tasks from the viewpoint of mobile and ubiquitous computing. With the help of CTT they manage to derive the dialog structure of a mobile human computer interface and then use probabilistic behaviour models to assign probability distribution over the activities space in order to infer the activity of a user. More concretely, they propose the usage of priority values assigned to each sibling in a node that are relative to the priority of all the remaining siblings. Then based on the model and the priorities, the probability of the transitions from the given state to the next is calculated. This is done based on the model history allowing for probabilistic reasoning over the user actions.

2.5.3.2 Requirements fulfilment

Below the requirements that are satisfied are discussed and the manner in which they are implemented.

Composition: CTT expresses composition in the form of a hierarchical structure where each root task has as leaves the actions, or tasks, it is composed of. Fig. 2.9 shows an example of such task where the composed action consists of the four actions A, B, C, and D.

Sequence: Sequences are represented by sibling nodes in a task tree with a relationship operator assigned between them. The sequential actions can have different relations (e.g. en- abling, disabling, order independence, etc.). In the example from Fig. 2.9, the actions A, B, and C are executed sequentially and have order independence, while action D is sequential to the last executed action and has the relation enabling.

Parallelism: Parallelism is achieved by the concurrency relation between two nodes (A_|||B), or by using the concurrency and information exchange operator (A_{|[]|B) where the nodes can} also exchange information while acting in parallel.

Repetition: CTT expresses an action repetition by simple assigning an asterisk sign to the repeated action (A_∗).

Choice and Priority: Choice in CTT is modelled by using the temporal operator for choice (A[]B). This indicates that both actions are executable, but when one of them is executed, the remaining one can no longer be executed. Priority in CTT is managed by the temporal operators thus, it is not available in the standard CTT formalism. However, Giersich et al. [58] extended the notation to use priority values based on which later the transition probability distribution was calculated.

Dependence, Enabling and Disabling: Enabling in CTT is handled by the enabling tem- poral operator (A >> B), which indicates that B cannot start before A was executed. It is also possible to use enabling with information passing with the temporal operator (A[] >> B) Sim- ilarly, disabling is modelled by the corresponding operator (A[> B) which indicates that A is disabled by B.

Interleaving: Interleaving in the sense described in the previous section cannot be mod- elled in CTT as composite actions have to execute the actions of which they are composed before another composite action is able to be executed. However, it is possible to use suspend- ing and resuming of composite actions to achieve the effect of interleaving.

Suspend and Resume: Suspend and resume are modelled by the suspend/resume temporal operator (A_{| > B), which indicates that A can be interrupted by B, and later when B is executed,} A can be resumed.

Synchronisation: In CTT synchronisation can be achieved by having an action enable the execution of two concurrent actions.

Independence: Independence is modelled by the order independence operator (A_{| = |B)} which indicates that the actions can be executed in any order, but when one of them starts, it had to be finished before the second can start.

Application-based requirements: The standard CTT notation does not support proba- bilistic durations. The same applies for observation model. However, Giersich et al. [58] have shown that it is possible to extend the model so that it can be used for generating probabilistic models that support probabilistic durations and observations. Giersich et al. have also shown that it is possible to apply CTT in its extended form to activity recognition problems.