Evaluation and Discussion

A smart lab infrastructure was developed with real-time sensing environment in two stages. At each stage, three main experiments were performed. The first experiment was conducted during the first stage where only binary sensors were used to conduct SPARQL based inferencing; details in section 8.5.1. The second experiment was also conducted in stage one to evaluate semantical segmentation algorithm and the results were presented in CHAPTER 3. The third experiment was conducted at the second stage of smart lab infrastructure development with multimodal sensors embedded within everyday objects to detect fine-grained actions of the users. The preliminary result of the third experiment results was presented in CHAPTER 4. Further development effort is required to complete the integration of single-user AR framework and multi-user AR proposed in CHAPTER 6 and CHAPTER 7. However, to illustrate the applicability of the proposed MSA and real-time multimodal smart kitchen environment, a typical multi-user scenario and AR steps is presented as a case study in section 8.5.2. Furthermore, open issues and challenges faced during the development of MSA and real-time smart environment are discussed in section 8.5.3.

8.5.1. SOA: Querying-based Inferencing Experiment

In experiment 1, the SPARQL based activity recognition algorithm presented in section 8.4.4.2 is evaluated with binary sensing environment depicted in Figure 8.12. The time duration between sensor activation and generation of inferencing results on the client device is measured to assess the overall system performance. The sensor activation time is only registered once the data are received by the web service to reduce factors such as network delays, time synchronisation between the sensing devices.

Table 8.2. User activity preferences with the associated total number of sensor objects

Activity

Number UAP Sensor Objects Sequence

Total no. of sensors 1 Make Indian

Tea

KitchenDoor1, KitchenCupboard1, TeaBagJar, IndianTeaSpiceJar, SugarJar, Kettle1, KitchenTap1, Fridge1, MilkBottle1, EatingSpoon1, Mug1

11

2 Make

Cappuccino Coffee

KitchenDoor1, KitchenCupboard1, CappuccinoBagJar, SugarJar, Kettle1, KitchenTap1, Fridge1, MilkBottle1, EatingSpoon1, Mug1

10

3 Make

Strawberry Juice

KitchenDoor1, KitchenCupboard1, JuicerMixerCup1, SugarJar,

KitchenCupboard2, ChoppingBoard1, Knife1, Fridge1,

StawberryPacket1, MilkBottle1, KitchenWaterTap1, GlassCup1, JuicerMixer1

13

4 Making

Chips And Beans

KitchenDoor1, FridgeFreezer1, ChipsBag1, KitchenCupboard2,

OvenTray1, HeinzBakedBeansCan1, KitchenWaterTap1,

MicrowaveBowl1, OvenDoor1, MicrowaveDoor1, CeramicPlat1

11

5 Make Pasta KitchenDoor1, KitchenCupboard1, PastaBag1, PastaPot1, KitchenWaterTap1, WoodCookingSpoon, PastaSauce, SaltBottle1

8

6 Taking

Medicine

KitchenCupboard1, MedicineContainer1, GlassContainer1, KitchenWaterTap1

4 Note: [sensor] - Changes in object(s) from previous activity

Table 8.3. AR test scenario types

Scenario Types Exact no. of Sensors Extra Sensors Activation Faulty/ Missing TP1 P O O TP2 O P O TP3 O O P

Table 8.4. Two examples of AR test cases

# Examples of tests specifications

1 TP1: #1,

TP2: #1, add KitchenCupboard2 and GlassCup1. TP3: #1, swap TeaBagJar and OvenDoor1.

2 TP1: #2,

TP2: #2, add KitchenCupboard2 and GlassCup1. TP3: #2, replace Mug1 with GlassCup1.

A fixed time window length is defined for six user activity preferences (UAPs) that are listed and tested with three different scenarios, see Table 8.2 and Table 8.3. The first scenario (TP1) activates the exact number of sensors defined in the user preferences, the second scenario (TP2) shows the activation of additional sensors objects, and the third scenario (TP2) shows a simulation of faulty sensors by using some sensor objects that are missing or not activated. The scenarios for the first two activities are illustrated in Table 8.4. Overall, each of the six activities is executed with three different scenarios by two actors (Exp).

The web service was deployed on the HP Z440 workstation with Intel(R) Xeon(E) v3 3.50GHz processor with 16GB RAM. The mobile application was tested on a Samsung S6 edge smartphone running Android 6.0.1 OS. The sensing data were collected using several touch sensors and door contact sensors using varied protocols.

The results in Table 8.5 indicate that on average, it takes 4477ms to receive the inferencing result on the mobile phone for all six UAPs with three different scenarios executed thrice. Overall, the results show little to no correlation between the number of sensors in the UAPs and the average time taken for inferencing and then communicating the results to the user.

Table 8.5. Results showing average activity inferencing duration from the last activities recorded

# Test Type Exp1 (ms) Exp2 (ms) Exp2 (ms) Avg. (ms) Avg. Per # (ms)

1 TP1 3890 3988 5127 4335 4472 TP2 5175 4176 4802 4718 TP3 4172 4145 4776 4364 2 TP1 4013 3953 4439 4135 4288 TP2 4131 4135 4725 4330 TP3 4275 4288 4630 4398 3 TP1 3926 3923 4353 4067 4411 TP2 4303 4316 4571 4397 TP3 5310 4225 4768 4768 4 TP1 4116 4175 4452 4248 4636 TP2 6330 4474 4695 5166 TP3 4410 4461 4614 4495 5 TP1 4150 4265 4409 4275 4584 TP2 4446 4414 5919 4926 TP3 4497 4533 4624 4551 6 TP1 4166 4801 4271 4413 4474 TP2 4532 4556 4563 4550 TP3 4415 4460 4498 4458 4477

8.5.2. MSA: Multi-user AR Case Study

In general, an activity is recognised by inspecting each action at two granularity levels, coarse and fine-grained. Figure 8.20 depicts a typical kitchen (S1) and living room (S2) ADLs actions conducted by Bob and Alice in a multi-room and single-room. At coarse-grained action level, general context, relations between ADL descriptions and user’s actions are used to assume an activity unfolding, i.e., cup (S7), kettle (S3), tea bag (S6), milk (S5), sugar and tap water observations for MakeTea ADL. Whereas, fine-grained level action detection method inspects deeper on how each action for a specific ADL is performed and determines whether the intention of a given action is satisfied. For instance, detecting “filling up” kettle from the water

tap, “pouring” water from the kettle into a cup and “drinking” from the cup when conducting

MakeTea activity.

In order to understand who is conducting actions for a given ADL in a shared environment, multi-user AR approach need to initially detect if the actions are occurring in multiple locations at the same time interval or assuming that a user cannot interact with more than two objects at a given time interval (T1-T3). Therefore, enabling parallel activities occurring in multiple rooms to be detected in T1 where Bob (S1) is filling up kettle (S3) in the kitchen and Alice(S2) is using TV remote (S4) in the living room (S2). Similarly, parallel activities in T3, drinking and turning the microwave on actions for different activities can be detected in a single room. The collaborative activity in a single room during T2 can be detected using discriminative sensors and approximating action association to a given user.

Room 2 Room 1 Coarse- grained Fine- grained Action Level T1 Pouring Filling up Drinking Single- user Multi- user Detect Identify Associate S2(Bob, Alice) S7(Bob), S8(Alice) Room 1 Bob(S1, S3)

Room 2 Alice(S2, S4) Room 1

Bob(S2, S3, S7)

Alice(S5, S6) Room 1 Alice(S2, S8)Bob(S2, S7) S1(Living) S2(Kitchen) Room 2 S2 (d1, d2) S3, S5, S6, S7 S1 (Bob) S2 (Alice) S2 (Bob, Alice)

S3/S7 (Bob), S5/S6(Alice) Single-room (Collaborative actions) Multi-rooms (Parallel actions) T2 T3 Room 2 S2 (d1, d2) S7, S8 S3 S3 _{S5 S6} S7 S3 S7 Single-room (Parallel actions) TVRemote S1 _S2 Ambient Object S8 S7 S7 S2 S2 S4

Figure 8.20. An example of multi-user activities in a shared kitchen and living environment

Apart from the technologies already mentioned in the previous sections, other supporting software components that are used to build the system are Jersey libraries [193] (i.e., Jackson library for JSON strings to java object mapping), Jena [265] Pellet [190](reasoner, see others [155]), Protege [159] (ontology editing tool), and Google API Services [266]) (i.e., for Text-To- Speech APIs, and Maps API. The Jersey library plays a key role in developing the RESTful web services for the function and parameter mappings of the incoming requests from the clients, as well as in producing and consuming data in various formats dynamically. In general, Jersey library is used to bind the web services with the Android application and mapping data into various object classes.

8.5.3. Discussion

The past system implementations with similar architectural styles and patterns to MSA have shown positive results in both functional and non-functional requirements; not only for AAL systems [267], [268]. However, finding suitable design patterns for a given application can be challenging and be easily misused [132], [134]. Nevertheless, several benefits of using a popular style and pattern exist. One example is system maintainability, which can improve code compensation level and efficient debugging for the developer. Furthermore, the decomposed MSA can enable any application to improve its scalability. In the case of the proposed system, additional multimodal sensing devices can be added within the SH so that the server can efficiently collect, process and disseminate data to multiple clients with minimal delay. Moreover, creating an opportunity to integrate other third-party services to extend the sensing capabilities of the smart environment.

Several lessons were learnt during the development of MSA and complex smart environment with open-source, off-the-shelf and bespoke sensing devices. Five key lessons learnt are as follows. Firstly, decoupling AR tasks with multiple web services can create additional overhead and duplicate codes. In addition, data analytics tools and hardware communications libraries require a large number of dependency libraries which are challenging to manage, outdated, not maintained, and not compatible with other libraries. Fortunately, microservices framework Spring book and dependency management tools such as Apache Ant, Apache Maven and Gradle are available to efficiently manage the dependency libraries and set up a new development environment more efficiently. Using these tools can prove to be more advantageous when integrating more third-party libraries and APIs to collect data from new smart home devices.

Secondly, developing bespoke sensing platform using IoT platforms or microcontrollers or using off-the-shelf sensing platform, it is essential to synchronise all the platforms to the same time server. This task can be tedious and challenging to achieve if the source code for the third-party library is not available. Nevertheless, data collected from the unsynchronised sensing platform can provide impact the accuracy of AR algorithms as the timestamp on the data can fall under different time interval or window-size. More specifically, impacting the ability to fuse multimodal sensor data from platform 1 compared to platform 2, which difference of even one minute in the internal clock. Moreover, re-programming/updating bespoke sensing devices with several modules deployed in the environment can be challenging to dismantle or unplug from objects physically. Therefore, over-the-air programming features available on some of the IoT based microcontroller should be utilised to avoid physical tasks.

Thirdly, low energy devices that are wireless generally have limited battery lifespan. Therefore, self-recharging capabilities using natural environments such as light, wind, heat and kinetic energy options should be explored to avoid replacing batteries every couple of hours or days. Moreover, a large amount of energy can be conserved when transmitting the data over a network to the cloud servers to perform some basic data analytics that edge devices are now powerful enough to compute themselves. Therefore, future work should explore edge and fog computing paradigms to not only reduce the energy but also increase the availability of the system.

Fourthly, a higher number of processing cores and optimised graphic-cards are required to support each of the AR tasks of data collections, storing, reasoning and disseminating results. Some AR tasks may require processing mathematical calculations and other more threads to complete the tasks. Therefore, intelligent thread prioritisation, scheduling and parallel processing are necessary to optimise and develop real-time AAL system.

Finally, the HCI with the system plays a key role in gaining further benefits from the system's capabilities. The system implementation uses a mobile application; however, our society is moving towards more natural and ubiquitous HCI. Other systems discussed [248], [250] in section 8.2, have already adapted the notion of augmented reality to overlay instructions on the camera or use natural gesture-/voice-based HCI. In comparison to the standalone/SOA SMART system and other systems implementations discussed in section 8.2, mainly have a web-browser based interface, this may limit the client devices from further utilisation, unlike with mobile devices with embedded sensor capabilities to collect meaningful and contextual data. In addition, instead of configuring additional dense or ambient sensors in the SH environment, more external sensors can be directly attached to a mobile device using any standard communication protocol [269].

8.6. Summary and Future Work

This chapter presents a literature review on system architecture styles and patterns adapted by recent AAL facilities and the challenges faced developing it. This was achieved by reviewing some of the latest studies and AAL system components that can complement one another. Based on the findings, a microservices-based architecture (MSA) approach is proposed for an AAL system. MSA integrates and extends the capabilities of the previous system implementations by introducing a light-weight, REST-based web service with an Android mobile application and web-browser interface. Multiple interconnected microservices plays a key role in interacting with the triplestore endpoint, collecting data from SH sensors and providing information to client applications. The service API web service provides activity inferencing and reasoning capabilities using Jena API; different reasoning engines can also be easily integrated. In