Relative orientation evaluation

The DARSIS system is evaluated in several environments to understand the effect of incorporating the relative orientation effect in the social interaction detection process.

Table 4.1: Confusion Matrices for the evaluation of the effect of Relative Orientation estimation for DARSIS system in percentages (%)

DARSIS Pos. Neg. Proximity Pos. Neg. Positive Negative 83 17 2.29 97.71 83 17 36 64

(a) Meeting room - 1

Positive Negative 60 40 8.14 91.86 65 35 60.57 39.43 (b) Meeting room - 2 Positive Negative 77 23 1.29 88.72 82.33 17.67 43.57 56.43 (c) Corridor

The system is evaluated in two situations: a) the overall DARSIS system and b) only the interpersonal distance estimation technique that DARSIS incorporates. The section starts with the evaluation methodology that describes the experimental setup and the performance metrics followed by the results of the evaluation.

4.5.1.1 Evaluation methodology

Section 4.2 highlighted the importance of incorporating the relative orientation is the detection of social interaction. State-of-the-art techniques do not consider users’ relative orientation, apart from [52] that obliges the users to restrict the wearing position of the device at a specific point, inducing a particular level of intrusiveness. An experimental analysis is performed to understand the effect of incorporating the relative orientation calculation in the social interaction detection process.

Several experiments were conducted in real-world indoor environments to evaluate the incorporation of relative orientation calculation in the social interaction detection pro- cess. Five participants in the age range of 25-30 and height range 1.65m-1.96m were involved while each participant received an off-the-shelf mobile (HTC One S) and was placed in one of the participant’s trousers pocket. The environments in which the ex- periments took place were meeting rooms and corridor. The participants were split in two groups of people, into two and three participant-interactions. Regarding the ex- perimental process, the participants placed the smartphones in their trousers pockets. The smartphones had pre-installed the DARSIS application and it was operating as a background service on the phone. After placing the devices in their pockets, partici- pants started walking for some meters in order to calibrate the relative orientation of

the device with respect to the user. Then, after the direction estimation has converged, the participants entered the environment in which the experiment would take place. Once the participants entered the environment in which the experiment would take place, they started to interact in two groups as mentioned before. In order ot establish ground truth, a human observer was employed that logged down the on-going social interactions. The groups in which the participants were interacting were changed after 10 minutes.

It should be noted that all participants were placed in close proximity, to force the social interaction detection to engage the relative orientation calculation mechanism. This will allow the understanding and quantification of the effect of the relative orientation calculation in the social interaction detection process. If the participants were not in proximity, then the interpersonal distance estimation technique would directly discard the detected participant from the social interaction detection process and classify them as non-interacting. At the end of all experiments, the produced data were analysed in order to infer the existence of social interaction first only based on the interpersonal distance and then including also the relative orientation.

4.5.1.2 Evaluation results

The initial performance metric used in the evaluation is the confusion matrices, which were calculated for both approaches, the overall DARSIS system and only the inter- personal distance estimation for interaction detection. The confusion matrices are presented in Table 4.1. The aim of incorporating the relative orientation calculation in inference process is to reduce the rate of false positives, which are the rate of social interactions that were detected but did not actually take place. Following the reduction of the rate of false positives would increase the overall accuracy of the system through the increase of the true negatives rate.

The initial evaluation showed that incorporating relative orientation estimation pro- duced high accuracy for the social interaction detection. The system managed 93.3% classification accuracy in the first scenario in the meeting room in contrast to the standalone interpersonal distance technique that managed only 69.7% of accuracy for

social interaction detection. Table 4.1 shows the improvement in the false positives rate because of the incorporation of the relative orientation. It should be noted that the in- terpersonal distance estimation technique is the same in both approaches. In the second scenario the system achieved 82.3% accuracy as opposed to the standalone interpersonal distance estimation technique that managed only 47.1% accuracy. This shows that an improvement of 33% in the overall accuracy was added by incorporating the users’ relative orientation estimation. In the corridor scenario, the DARSIS system was able to detect correctly 85.2% of the social interactions and the standalone interpersonal distance estimation technique achieved only 64.2% accuracy. This provides another indication about the significance of incorporating the relative orientation calculation in the social interaction detection process. The incorporation of the relative orien- tation calculation in the social interaction detection process showed an improvement of at least 20% in the overall accuracy in comparison to the standalone interpersonal distance estimation.

The classification rates of each approach are presented in Table 4.1, to provide a better understanding of the effect of the relative orientation calculation in the social interaction detection process. As shown, the true positives rates are the same in both approaches and did not receive any improvement. This is because in the social interaction detec- tion process, the interpersonal distance estimation technique has the dominant role. It should be noted that the incorporation of the relative orientation calculation mecha- nism induces a small amount of error. This error is due to processing and computation error of the algorithm but also due to magnetic disturbance in the environment. Ta- bles 4.1b and 4.1c quantify this error through the reduction of the true positives rates. The largest error around 5% is observed in Table 4.1c due to the relative orientation algorithm. However, while observing also the 42% decrease in the false positive rate followed by similar increase in the true negative rate, the 5% error in the true positives is a trade-off that is worth taken. Due to the dominant factor of interpersonal distance estimation, the false negative rate can only be reduced by the relative orientation de- tection. The false negative rate was not affected by the relative orientation calculation according to Table 4.1a. Tables 4.1b and 4.1c show a minor increase of at most 6% in the false negative rate due the relative orientation estimation, including the facing

direction algorithm and the magnetic interference of the environment.

The aim of this evaluation was to show the improvement in the accuracy of social interaction detection through incorporating the relative orientation computation but also the robustness of the system in various environments. Table 4.1 shows that the aim of reducing the rate of false positives was achieved from 33% to 42% improvement in the accuracy, while also led to the increase of true negative through 33% and a 97.7% accuracy for the true negatives rate due to the incorporation of the relative orientation calculation mechanism.