object? Who is
7.5. Testing and Evaluation
An ADL scenario is described in Figure 7.5(a) where three mixed activities are carried out in the shared kitchen by two users, Bob and Alice. The actions for three activities, MakeTea (A1),
MakePasta (A2) and MakeToast (A3) occurring between 10.00am to 10.03am are illustrated.
The sensor observations are collected by the respective event handler classes in SensingUtils package of the web service and appended to the observations queue. Each sensor observations occurring at a given time (tn) are then semantically segmented based on the objectβs relationship
with a set of actions specified in the ADL description and only appended to the activity (An)
thread if the observed action matches the ADL description.
The πβπβ algorithm is performed in four stages by the individual activity thread and the sample results are depicted in Figure 7.5 (b). The first stage is to perform context analysis of each activity and calculating ππβ, i.e., identifying the location, key objects and time interval to calculate the confidence level of the activity occurring. The location information of the everyday object is predefined for fixed objects such as kettle, toaster and microwave. The key objects for each activity, location and time interval are mapped with the importance of a given activity which is stored and queried from the triplestore.
fs3: alice X, Y, Z 1 0 fs6: alice st1 to1 1 0 to3 to3 X, Y, Z X, Y, Z st3 st3 X, Y, Z fs1:bob fs6: alice Timeline tn st1
(A) Activity Segmentation An
1 0 to2 to2 X, Y, Z st2 st2 X, Y, Z fs4: alice fs5:bob 1) CCL 2) FCL 4) MAR (B) Segmented Observations Analysis
A1: MakeTea 96.67% st1=true (pouring) 50% alice (x1), bob (x2) Other (x2) A2: MakeBaked Beans 91.67% A3: MakeToast 93.33% alice (x2), bob (x1) alice (x2) other (x1) st2=true (transfer) 20% st3=false (spreading) 0% 3) ARCL 61.67% 37.92% 23.33% fs2:bob
10:01am 10:02am 10:03am 10:00am
to1
fs5: alice
Figure 7.5. (a) Three ADLs segments processed by ARCL algorithm, (b) four stages of multi-granularly single and multi-user activity detection result.
The second stage is to inspect sensor data to detect if the user has performed fine- grained actions such as βpouringβ by inspecting accelerometer, gyroscope and liquid level sensor data. The threshold to detect βpouringβ action vary depending on the dimensions of everyday objects and the quantity of content inside. Therefore, thresholds are predefined for when liquid quantity is low, medium, and high along the degree of rotation/tilt position for each object type. The associated importance values of both stages are used to calculate ππβ and β±πβ. The final stage is performing β³πβ using fingerprint sensors and associating sensor observations to the user identified. In addition, other sensors attached to the same object to the fingerprint sensor is grouped and associated with the user.
7.5.1. Discussion
Despite the scalability and deployment challenges to attach a fingerprint sensor to each everyday object, this approach can identify individuals more discriminatively than passive identification (ID) broadcasting based approaches[235], [236]. For instance, smart clothing with
passive RFID tags [237] can be worn by another person or incorrectly assigned and Bluetooth based smart beacon deployed in the environment that are read by the smartphone belonging to another individual. However, RFID tags and beacon are very unobtrusive and passive sensing approach to detect the number of users and triangulated locations [238] in a shared environment and assume the link to a specific user.
One of the limitations of this approach is that each everyday object of interest would require at least one fingerprint sensor in order to associate each sensor observation with a given user. In addition, the traditional capacitive fingerprint sensors can only cover the small area where a userβs fingerprint can be scanned; hence, the position of the sensor is important. However, in the recent advancement in ultrasonic fingerprint technology can help overcome these limitations and reduce the cost of the sensors. Ultrasonic fingerprint sensor technology has been under investigation for more than a decade to overcome the poor performance of capacitive fingerprint sensors when fingers are oily, wet and it can easily be spoofed using printed or moulded fingerprint images [239], [240]. Recently, Qualcomm announced advance fingerprint scanning and authentication technology capable of covering a larger area of the display, thick glass and metal surface [241], [242]. In addition, detection of directional gestures, heartbeat and blood flow even when immersed underwater can be used to add layers of authentication and identification of a user. Mobile phone manufacturer such as Vivo has already integrated this technology into their flagship phones and others such as Apple, Samsung, Xiaomi, and OnePlus 5 are expected to follow soon.
Another limitation when adapting a dense sensing approach is that perishable and recyclable items such as soap, plastic bottles and other packaging materials pose scalability, reusability and integration challenges. In addition, the design of the everyday items and size dimensions parameters determine the sensor positions, hence, the varying threshold values.
7.6. Summary and Future work
This chapter developed the course- and fine-grained activity recognition (AR) algorithms and estimates AR confidence level (πβπβ). The coarse-grained confidence level (ππβ) algorithm extracts location, time and key objects for a given activity along with their respective importance levels from the segmented sensor observations. Each key actions and parameters are given a pre-defined importance value based on the degree of belief for the action required to occur for calculating the confidence level. To recognise granular user actions using a given object, i.e., βpouringβ water from the kettle to cup, the fine-grained confidence level (β±πβ) algorithm is introduced which analysis the sensor observation against the threshold values predefined with the importance level information. The sum of all fine-grained actionβs
importance values is considered three times more important than the ππβ value when calculating the overall πβπβ.
In addition, Multi-user AR (β³πβ) algorithm is proposed, which can detect, identify and associate actions of several users performing collaborative or parallel activities. The approach leverages fix time windowing process to detect maximum objects interactions with a pre- defined threshold and multi-location events. Moreover, smart textile with RFID tags and fingerprint sensors attached to everyday objects is used to identify and associate sensor observations to users. However, the key limitation of this approach is the scalability and maintainability challenge to integrate fingerprint sensors in every object wirelessly. The layered microservices-based system architecture (MSA) system and key sensors have been proposed to create a multi-user smart environment. The key sensors include ambient sensors (door/window, PIR, UHF RFID reader) and dense sensors (inertial measurement unit (IMU), fingerprint, RFID tags, and liquid level sensors) for a non-invasive and non-obstructive data collection. The approach is applied to a use case application scenario where mixed kitchen-based activities with multiple users performing collaborative tasks.
The future work will involve implementing and evaluating the performance and accuracy of the proposed πβπβ algorithms with β³πβ. In addition, optimising AR performance and investigating in activity learning techniques to evolve ADL models.