Cooking task

The second use case is a cooking problem where a single user is preparing a meal, having lunch and then washing the dishes. The objective is for the user to successfully complete a typical cooking task problem.

2.3.2.1 Motivation

The cooking problem is a Kitchen Task Assessment (KTA) problem [14], where the aim is to detect whether the person is executing the task in the correct order, and if we stretch it further to an assistive problem, the system would like to detect inaccuracies in the user behaviour and assist her in correcting her mistakes and successfully achieving the task objective. This problem is a typical Activities of Daily Living (ADL) problem and it has applications where the user could suffer from Alzheimer or similar diseases but the disease is still in early stages and she wants to preserve her independent lifestyle for as long as possible [128]. The solutions of such kind of problem are extremely important in Europe where the elderly population is increasing while the birth rate is decreasing [38, 13].

The cooking task is also a typical single user behaviour and thus uncorrelated, where the agent’s actions do not depend on other users’ behaviour but where she interacts with the sur- rounding environment, thus making her behaviour dependent on the environmental factors. It is also a problem where there is a lot of interaction with different objects. This requires modelling the problem on a more fine-grained level compared to the meeting scenario.

The reason for choosing this particular example is that it presents a simple cooking recipe, with just a few ingredients, yet it possesses all basic interactions with the environment taking place in a cooking situation (manipulating objects used in the everyday’s tasks in the kitchen, using the cooking appliances, setting the table, eating, and cleaning). This allows us to explore the complexity of a cooking task and a real world single user behaviour without running into unnecessary details of complex meals.

2.3.2.2 Problem description

A person is cooking a carrot soup in a kitchen supplied with the necessary kitchen appli- ances. The person starts by washing her hands, then cutting the carrot, putting it into the pot and cooking it. Later after the meal is ready, she serves it in a plate, puts water in a glass, and sits on the table to have a lunch. Finally, after the person has eaten and drunk water, she stands up, goes to the sink and washes her utensils. There are generally the following experiment stages that should be executed in the same order.

wash hands -> prepare to cook -> cook carrot soup -> serve meal -> eat and drink -> clean up

On the other hand, the intermediate actions that take place (such as fill plate, fill glass, move, etc.) can be executed in any causally correct order. The locations in the kitchen are sink, counter and table, which are locations that could be reached only by walking from one place

to another. Additionally, there are fixed locations, or places, which from a certain locations can be reached only by moving the hand. The places are cupboard and oven, which could be reached from the counter. Furthermore, different objects with varying functions and properties are used: cutting board, pot, plate, glass, bottle, knife, spoon, sponge, and the additional water and carrot.

2.3.2.3 What is to be modelled?

As the cooking task involves a lot of fine-grained activities, it is a more complex problem than the 3-person meeting (regardless of the fact that in the latter we have people acting in parallel). Thus also the behaviour and environment elements are much more than in the meeting scenario.

Table 2.3: Elements of the cooking task behaviour and environment. Here the number of users, actions, objects, locations, and places as well as the initial and the goal state are described.

Types of elements Instances Users one default user

Actions wash, wait, move, take, cut, put, turn on, cook, turn off, open, fill, close, sit down, eat, drink, stand up Objects cutting board, pot, plate, glass, bottle, knife, spoon, sponge, water, carrot

Locations sink, counter, table Places cupboard, stove Initial state the user is at the sink

Goal state the user is at the sink after cooking, eating the meal and washing the dishes Execution length about 80 actions that need to be executed in order to reach the goal

Table 2.3 shows the elements that are to be modelled. It can be seen that there are 16 actions that can take place, and most of them (except for wait, move, sit down and stand up) involve manipulating any of the given objects. Additionally, there are 10 objects in the environment that are manipulated by the user that indicates the need of a mechanism to cope with a high number of different choices the model will be faced with (See Fig. A.1 in Appendix A for the locations of the objects throughout the task).

2.3.2.4 Sensor Infrastructure and Datasets

## l o c a t i o n s ###

sink mov ing c o u n t e r table

0.0 1.0 0.0 0.0

0.0 0.0 1.0 0.0

## fixed pl aces ###

sink mov ing c o u n t e r stove c u p b o a r d table

0.0 1.0 0.0 0.0 0.0 0.0

0.0 0.0 1.0 0.0 0.0 0.0

## o b j e c t s ###

carr ot knife c _ b o a r d pot w _ s p o o n plate glass bo ttle spoon s ponge

0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0

1.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0

Figure 2.3: Example observations from the cooking task dataset. Here the observations are either 0 when no sighting at this location / place / object was observed, and 1 when a sighting was observed. One can observe the locations, the places and / or the objects.

Fig. 2.3 shows an extract of the observations from the cooking task dataset. The first row shows the observations for the locations with 1.0 indicating that the person has been observed

at the given location. The second row shows the observations for the fixed places. The last row shows the observations for the objects being observed. Seven cooking tasks were recorded. Each of the tasks lasted about 7 minutes and although the task at hand, namely cooking, was staged, the behaviour of the participants while achieving their goal was left to themselves. This resulted in different execution paths leading to the goal state and increased the model variability needed to be able to recognise the correct behaviour. Additionally, the datasets contained between 636 and 1207 samples.