Experiment 3: Comparison with learning techniques

In this experiment, we compare our evidence decision network results with two class machine learning techniques, Naïve Bayes Classifier and J48 De- cision Tree. The purpose of this comparison is to examine whether we can get comparable results using the evidence decision network to learning tech- niques, as a ’sense check’ that our evidence theory-based approach can per- form reasonably in comparison to other recognition approaches. The experi- ments were run in two cross validation modes (1) using limited training data (one third, cross validated) and (2) using a ’leave one day out’ cross validation approach. When limited training data was used, the evidence decision net- work with temporal extensions clearly out performed both Naïve Bayes and J48 as shown in figure 6.12. Both evidence decision network approaches in- corporate domain knowledge so their performance on limited training data is not significantly improved on the ’leave one day out’ technique. The learning techniques, on the other hand, rely totally on training data for their results, performing much better on the ’leave one day out’ approach than on limited training data, as shown in figure 6.12.

It is interesting to compare the evidence decision network results of limited training data (with greater reliance on domain knowledge) with the learning results from the more comprehensive leave-one-day-out training technique, as per the bolded figures in table 6.7. An evidence based approach will typically be used when training data is limited or not available, so the results from lim- ited training data are encouraging - the technique with temporal knowledge exceeds the learning techniques, and without temporal knowledge still per- forms well compared to learning techniques. Since evidence theory is suited to the incorporation of domain knowledge, this result is encouraging. In particu- lar, temporal knowledge of absolute times and transitory evidence for endur- ing situations can be incorporated and used to improve recognition accuracy. Evidence theory will be useful when training data is not easily available and where domain knowledge can be gleaned from expert knowledge and user knowledge. These sources can be used to obtain inference knowledge in a piecemeal approach, with users providing information on absolute times, sit-

0 0.2 0.4 0.6 0.8 1

leave house use toilet take shower go to bed prepare breakfast

prepare dinner

get drink

No time EDN Temporal EDN Naïve Bayes J48

Situations

Figure 6.11: Comparison of F-measure between Time extended Evidence, Naïve Bayes and J48 Decision Tree with one third training training data

0 0.2 0.4 0.6 0.8 1

leave house use toilet take shower go to bed prepare breakfast

prepare dinner

get drink

No time EDN Temporal EDN Naïve Bayes J48

Situations

Figure 6.12: Comparison of F-measure of time extended evidence, Naïve Bayes and J48 using Leave One Day Out cross validation.

EDN without time

EDN with temporal

Naïve Bayes J48 Decision Tree

One Third Training 0.40 0.68 0.49 0.34

Leave One Day Out 0.45 0.70 0.58 0.51

Table 6.7: F-measure comparison of Time extended evidence, Naïve Bayes and J48 for one third and Leave One Day Out cross validation

uation descriptions and durations, and experts providing knowledge of sensor mass functions and sensor quality.

6.3.4

Experiment 4: Comparison with published results

Van Kasteren [109] and Ye [119] have both published inference results based on using a ’leave one day out’ cross validation technique. Van Kasteren evaluated Hidden Markov Models (HMM) and Conditional Random Field (CRF) recog- nition techniques on the data set. Each technique was tested in three sensor representations:

• raw sensor representation which returns a 1 when a sensor fires;

• change point sensor representation which returns a 1 when a sensor read- ing is changed;

• last observation sensor representation which returns a 1 if a sensor con- tinually fires, and gives a 0 when a different sensor fires.

Van Kasteren et al. only published the raw sensor representations in which all the sensor values are only 1, so we compare with raw sensor representation results only. To measure accuracy, they use a class accuracy measure calculated as average percentage of correctly recognized timeslices per situation:

class = 1 C. C X c=1 ( PN

n=1(inf erredc(n) = truec(n)

Nc

)

(6.5)

N is the total number of time slices, C is the number of classes and Nc is the

total number of time slices for class c. We calculated van Kasteren et al.’s class accuracy measure calculation for our time-extended evidence using the same ’leave one day out’ cross validation approach as Van Kasteren. A summary of the comparative results is shown in table 6.8.

Ye Lattices evidence decision network with temporal van Kasteren HMM van Kasteren CRF 88.3% 69% 49.2% 44.6%

Table 6.8: Comparison of class accuracy of temporal EDN with published re- sults from Ye [119] and van Kasteren [109]

Ye’s situation lattices approach yields a class accuracy of 88.3% using raw sen- sor representations and the ’leave one day out’ cross validation technique. This is higher than the results from the temporal evidence framework (69%) and van Kasteren et al.’s HMM results (49.2%). Ye’s lattice method includes ab- solute time in the inference method, and combines both training and domain knowledge. However, timeslices in which no sensor changes take place are ex- cluded in Ye’s results, but are included in the evidence decision network and in van Kastersen’s work. These ’inactive’ timeslices are hard to infer because of the lack of sensor information so the data set is likely to yield improved results to some degree.

HMMs consider the sequence in which situations occur. They do not consider absolute times (unless explicitly captured in the training data) or durations of situations. The relative performance of HMMs and CRFs for the van Kasteren data set are explored further in [109]. Van Kastersen achieves an accuracy of 79.4% using a richer sensor representation, ’changepoint plus last sensor’ representation as described in [109].