or situations occurring at a particular point in time by interpreting available knowledge. This knowledge may come from a range of sources, both physical (for example, tem- perature or motion sensors), and virtual (for example, calendar schedule data or web browsing history).
In this section, we overview research challenges in the field of situation recognition. The first two are central to the work described in this thesis: Capturing the structural semantics of knowledge (Section 1.2.1), and making the decision process intelligible (Section 1.2.2). We discuss these in more detail before briefly overviewing other research challenges in the field (Section 1.2.3).
1.2.1
Capturing the Structural Semantics of Knowledge
The use of multimodal sensors produces datasets with heterogeneous features, differing in format, structure, scale, and frequency. Chen et al. (CHN+12) identify that the major- ity of published work that involves the capture and storage of such sensor information tends to adopt the use of ad-hoc data structures.
Beyond sensor data, many pervasive applications make use of higher-level encod- ings, such as ontologies, to describe application level concepts, such that sensor data can be mapped to the data model and annotated with additional semantic mean- ing (RMCM03; CFJ04); for example, indicating that a motion sensor is mounted on a particular door, or that a temperature sensor value corresponds to the conceptwarm. Such an encoding, which provides developers with a symbolic model of their software’s operating environment, captures information used to characterise the situation of entities in the environment (theircontext (DA00)) and is commonly referred to as acontext model.
The capture of expert knowledge can contribute to making inferences that cannot be obtained directly from sensors. For example, the modelling of containment relationships between physical locations supports the inference of a person’s located in a larger space when their presence is detected in a space it encloses.
This ability to generalise knowledge can contribute to situation recognition. However, ontological models in the literature are typically ad-hoc, primarily designed to meet the needs of particular applications or types of knowledge (SKDN09).
ging (CBB+12; SHS08), it is not yet the case for these higher-level models. Without their existence, it becomes that much more difficult to share and reuse data and data models for situation recognition across applications, platforms, toolsets and research- ers (SKDN09; YSD+12).
1.2.2
Making the Decision Process Intelligible
Intelligibility concerns the ease with which the decision making process of any recogni- tion technique can be understood. This ability is particularly important when there is significant error in recognition accuracy, the cause of which needs to be diagnosed. In such cases, a lack of transparency can present a challenge.
Situation identification techniques are typically classified into two main categories: Knowledge-based, where inferences are drawn from rules specified by experts, and learning-based, where machine-learning techniques discover relations between sensor- data and situations (YDM12; CHN+12). It is generally the case that, at best, the decisions of learning-based techniques are not as intelligible as those of knowledge- based techniques, and, at worst, opaque. However, machine learning techniques usually outperform knowledge-driven techniques due to their ability to better handle noisy sensor data.
As the field of situation recognition has matured from constrained prototypes to real- world systems, the general research trend has seen a move away from knowledge-based models towards learning-based models. However, the cost of this move has been the loss of ability to scrutinise the decision making processes, which we have already identified as useful.
The challenge is to develop recognition models that harness both the power of learning to attain high accuracy, while retaining the ability to inspect and understand the decision making process.
1.2.3
Other Challenges
Situation recognition faces a number of additional challenges, that, although important, are not the focus of this work.
Unsupervised Learning As described above, situation recognition techniques depend on the availability of high quality training data. While the majority of works require that this training data be labelled with a complete record of the activities that
have taken place, there is a research trend towards the development of models that rely only on partial annotations, or no annotations as all. (HMJ+09; GCTL10; BVMR06; YS13; YSD14a; SR06; SS09). Developing these techniques to the point where their accuracy matches the levels attained when fully annotated training data is available is an ongoing challenge.
Recognition Complexity The majority of work in the literature focuses on single- subject scenarios involving situations that occur largely in sequence. However many situations involve interactions between multiple people, or situations that co-occur or are interleaved (GWW+09). Although some recognition techniques have targeted this area (MBK08; HY08; HNS11a; SZC11; YS13), significant additional research is warranted.
Automatic Error Detection and Correction Once a recognition system is set- up, its performance can degrade for many reasons, including sensor-drift or introduction of noise, sensor-malfunction, or even changes in the way a situation is realised. While some work has been published in this area (FD13), approaches to automatically correct errors or adapt the recognition process in response to drift in sensor readings remain largely unexplored.
Detecting Anomalous Situations Another challenge to be addressed is the de- tection of unexpected or rarely occurring situation, the nature of which makes them difficult or impossible to train for (CHN+12; YDM12), or situations with duration so short as to make their recognition difficult. Examples include health problems such as a stroke or heart attack, or an event like a home burglary.
Supporting Transfer Learning Training data is required in order to learn a situ- ation recognition model and evaluate it. However, the process of collecting such data is often costly in terms of time and effort required to annotate it withground truth—a description of the situations occurring, along with the time they occur. Consequently, an important target for research in situation recognition is the ability to learn a model in one environment, for which training data is available, and successfully apply that model to an unseen environment for which no, or little, training data exists. This task, known astransfer learning, alleviates the need to collect significant amounts (or ideally any) training data from the new environment, and reduces the expertise required to install and initialise the recognition system.