Abstract— Over the past few decades, the concept of Internet of Things (IoT) has become increasingly popular in the research community for its promise to homogenize the heterogeneous pool of devices having sensing and actuation capability, which are increasingly getting internet ready. On the other side, the need for a social companion to the human kind in form of a humanoid robot, which is acceptable to the society is another long-standing challenge waiting to be solved in a technically feasible and creative manner. So, in the present research, we have tried to address this challenge by creating a personal assistant out of the NAO robot by enabling it to perform the various tasks that its owner would require of it in a way customized to each of its owners. In this process, we have proposed a novel architecture through the IoT paradigm by using a demand-pool based brokerage system for actuation on the robot through the internet and an asynchronous polling based I/O system to trigger various events on the robot as per the owner’s requirements. The results and performance of the robot through this framework have been discussed and future guidelines have been provided to make this software more feature-rich and learning oriented.
Key words: Humanoid robot, NAO Robot, Internet of Things, service oriented architecture, personal companion, human-robot interaction.
I. INTRODUCTION
Human kind has always strived for a constant companionship in their social life, be it in form of friends, family members, colleagues or some random acquaintance in his fast paced life. This eternal pursuit of a constant friend has led the robotics researchers come up with a robot which would be morphologically similar to the human and thus the humanoid robots came into existence. However, the challenge that inspired us to do this research was to make a generic framework for humanoid robots through which we can implement the necessary functionalities and requisite intelligence so that the robot can act as a personal companion of the owner. In the process of solving this challenge, we realized that, in general, there will be a constraint on the processing and storage capability of the robot. Hence our main approach was to develop a system which would require minimal onboard storage and processing and the control commands would be sent to the robot to initiate the appropriate behavior at the proper time.
In order to realize this framework, we tried to develop an architecture for the application to run onboard the robot and the application server simultaneously, as well as devise some
Soumyarka Mondal and Gora Chand Nandi performed this research at the Robotics and Artificial Intelligence Laboratory, Indian Institute of Information Technology Allahabad.
Soumyarka Mondal graduated from the Institute in the area of Information Technology in 2016 and is currently employed by Morgan Stanley Advantage
kind of triggering mechanism on the robot to initiate the scheduled events which would be defined by the user. Every individual user would have his/her own distinct life style and thus, to capture this pattern of personal habits, the application would need to have a scope for customization of the various services offered by the robot to the user. Hence, the main aim for developing this platform was to make the life owner easy by enabling its robot to perform various daily chores in and around the household. So, to achieve the same, we have devised a novel demand-pool based architecture (on which the application has been modeled) where all the services being offered by the software would be available on the global application server and the robots would have to subscribe to these services using their unique identification. Since, each of these robots would be mapped to some user account, hence this entire application would be totally open for customization by every user based on their personal preferences. The main aim behind the development of this software was to create a generic application test-bed using a novel architecture based out of the Internet of Things (IoT) which could be used for developing distinct modules for enabling a humanoid robot to act as the personal companion of the owner.
The paper has been arranged in the following manner. In the next section the basic idea behind the personal companion in form of a humanoid robot has been described followed by the behavioral analysis of this framework. In the subsequent section, we have discussed about a novel architecture using the IoT where the application runs simultaneously on the robot and the application server. Subsequently, we have described the details of our implementation of the software using the various open-source application programming interfaces (APIs) and the consequent protocols adapted for necessary communication between the robot and these cloud-based APIs. In the fifth section, the performance analysis of the NAO humanoid robot running our application has been done with regards to the various APIs used, which is then followed by conclusion and discussions.
II. THE PERSONAL ROBOT
There have been a number of studies [1, 2] where the human psychology was studied through various surveying techniques regarding the nature of humanoid robot that would be most suitable in the household. The outcome was that most of the participants in the conducted survey saw the humanoid robot in the role of a potential companion, who would help him in his daily household chores and other needs. A few of the participants also envisioned the robot to be a friend,
Services India Pvt. Ltd. as a Technology Associate (email- [email protected]).
Gora Chand Nandi is currently the Dean of Academic Affairs at the Institute (email- [email protected]).
Personal Robot: Towards developing a complete Humanoid Robot
Assistant using the Internet of Things
however, the normal chores were given preference to over the child care, animal care or such other critical tasks. In those studies, it was also observed that the human-like physical and morphological structure was desired for better adaptability within the household and human-like communication was desired from the robot. All these observations in terms of human robot interaction were noted while deciding the framework for the application which primarily meant to make its way into the homes of individual users as a daily companion.
Figure 1 SoftBank Robotics NAO Humanoid Robot
Keeping in mind these particular requirements, we chose the NAO [3] humanoid robot from SoftBank Robotics as the robot to build our application on. The NAO humanoid had the requisite appeal to make it acceptable in the human society as a household member [4]. It is basically a computer with multiple actuators, sensors and various interfaces for communication. It runs the NAOqi OS onboard which is a version of the Gentoo based Linux system. A short chart on the various specifications of the robot has been mentioned below:
Item No. Topic Specification
1. Height 58 cm
2. Weight 4.3 kg
3. Power Supply Li-Ion 48.6Wh
4. Degrees of
Freedom 25
5. Intel Atom 1.6 Ghz
6. Built-in OS NAOqi 2.1 (Linux based)
7. Programming
languages Python, C++, Java, Matlab, Urbi, .Net
8. Sensors Two HD Cameras, four
directional microphones, sonar rangefinders, two infrared emitters and receivers, inertial measurement unit, nine tactile sensors and eight pressure sensors
[image:2.595.69.287.457.665.2]9. Connectivity Ethernet, Wi-Fi Table 1 Specification Sheet of the Nao Humanoid
The main inspiration behind this framework was to transform the NAO humanoid into a “Personal Robot”, one which would cater to the various needs of its owners in the household. Now we have seen that in this current age of
ultra-connected social life, most of our tasks are somewhat dependent on the internet either for fetching some information or to store something for future use. This observation led us to exploit the existing elements of the TCP/IP suite and create our own service-oriented architecture based application which would connect the NAO to the global application server and thus it would serve as a personal cloud-connected companion through the Internet of Things.
In order to enable the robot to serve as personal companion, the daily life of a typical owner has been studied and it was observed that for optimal help from the robot the daily routine should be divided into four segments, viz.: Early morning brief, Throughout-the-day activities, Leisurely evening and Daily night briefs. The various behaviors developed for each of these segments throughout the day have been discussed in the next section.
III. BEHAVIORAL ANALYSIS OF THE APPLICATION
Being inspired from the daily routine of an average human being, the different behaviors developed for the application have been categorized as below:
A. Early morning brief
In this phase, the robot initializes itself and then fetches the alarm time set by the user. Next up, upon receiving voice command from the user, the robot delivers his daily morning brief, which comprises of the latest news, weather updates and e-mail and other communication related notifications. Each of these individual behaviors are fully customizable by the user who sets his preferences in the application server by logging in through its web based Graphical User Interface (GUI).
The news briefing behavior uses the various RSS feeds from the leading online news agencies and the robot then communicates the corresponding headlines to the user. An additional feature for customizing the categories of news to be received has also been provided in the GUI and is open to customization by the user.
The weather briefing behavior briefs the user about the current weather conditions of the region and also does a daily forecast of the precipitation conditions. Again, the concept of IoT has been used here to geo-locate the robot using its global IPv4/IPv6 address and thus it automatically gives the user the correct weather details for the appropriate region.
Upon receiving further voice commands, the robot then communicates to the user about his e-mail or any such instant messaging (IM) notification and thus keeps the user abreast about his online communications since last night.
B. Throughout-the-day Activities
Some of the behaviors have been designed in a way such that they will keep on running in the background services of the robot throughout the day. These include the tasks of remote home surveillance, giving company to the children and the elderly in the home and also engaging in intelligent conversation with the other members of the household.
object tracker mode and if any activity is noted, it sends a screen shot of the same to the pre-defined cell-phone and the email of the user.
The NAO has been programmed to tell stories and such other activities, so that it can keep the children busy in the household. Also, in some special cases, this module can be tuned for the purpose of assistive robotics through which the NAO can help the children with autism to learn about their surroundings in a friendly manner. It has been shown in studies [5, 6, 7 and 8] that children suffering Autism Spectrum Disorder (ASD) can meaningfully interact with this fully programmable humanoid and NAO can actually help them learn about basic social interaction and communication skills. The NAO has also been programmed to engage in meaningful conversation with the members of the household. There are various trigger phrases like “Hello, NAO”, “Hi NAO” etc. which would make the NAO go into the conversation mode.
C. Leisurely Evenings
These behaviors have been designed to help the user relax at home after a tiresome day at work. These include the behaviors of streaming the music from the playlist custom created by the user, attending to guests coming to the house etc. Through these behaviors, the NAO robot enables the user to repose at his home, either through attending his guests or through creating an environment in his home according to his mood. These voice-activated entertainment choices have been synced through the user’s own choices pre-defined to the robot.
D. Daily Night Briefs
Before going to sleep, the user can set various notifications on the NAO to help him remind the next day or as desired by the user. These notifications can be of various types, either to-do-tasks, some bill payment requirement etc. The robot will then upload these tasks on the global application server to enable it to serve these notifications at an appropriate later time. However, when the time for these notifications are triggered and the robot is offline, then the user can opt to send the notification to his cell-phone and email directly from the application server.
Hence, the entire application framework has been designed to take an inspiration from the daily life of the normal human household and help the owner as an accompanying companion in whatever capacity necessary. However, in order to fully realize this application, the existing elements of the Internet of Things (IoT) were used to create a suitable architecture on which this application was built upon. The following section discusses this architecture in details.
IV. APPLICATION AND NETWORK ARCHITECTURE
Since the basic inspiration behind this application is to make a Service Oriented Architecture (SOA) based collection of services, it was necessary to derive our own set of protocols for message passing and remote actuation on the robot. As defined in this book [9] by Microsoft® Developer Network, that SOA is defined by “A loosely-coupled architecture designed to meet the business needs of the organization” [9], our application basically needed a secure mode of
communication with the robot and an intuitive graphical user interface (GUI) with which the owner can interact with the software and decide to turn the services on or off on his robot or even fine-tune them.
[image:3.595.305.531.189.358.2]The message passing and communication flow of the entire application has been summed up in the following figure.
Figure 2 Data and Control Flow in the Application
Basically, the application consists of inter-relations between the four entities i.e., the global application server (GAS), the NAO humanoid robot, the end-user (EU) who will actually be interacting with the robot and various other devices (OD) of the user which are network-enabled and are connected to the home network either through Wi-Fi or through Ethernet. The roles and functionalities of these four entities are described below.
The GAS will basically act as the server which will coordinate the functioning of the robots in their respective households. Each of these robots will be uniquely identifiable to the server with their unique robot id (RID) and will be mapped with the individual user ids. These user ids will be used via a Google® account for accessing the Google APIs with proper customization at the user-level. All the services offered to the user will be having a unique application id (AID) and whenever the robot will communicate with the GAS, it will send the RID and the AID in the request header. Also, for the purpose of triggering remote events on the robot, the GAS will send the corresponding AID to the robot on its last known IP address. The procedure for remote event generation on the robot has been discussed in details in the following sections where we discuss the role of the other connected devices (OD) in greater depth.
completely hard-coded into the NAO robot. Now, these simple and elementary behaviors have been combined to create more complex behaviors which would appear more natural to the end-user, who wants nothing more than some intuitive form of communication with the robot. The way in which we have incorporated this architecture in our application running onboard the NAO has been shown in the Fig. 3.
Figure 3 Proposed use of Subsumption Architecture in the software
There are a group of elementary behaviors such as movement in the world by avoiding obstacles, maintaining a minimum battery charge of greater than 20% etc. These lower level behaviors are very much self-achieving in nature and are more or less independent of each other. By loosely coupling these elementary behaviors, we can create somewhat more effective tasks like maybe we can combine the movement module with the obstacle avoidance module and thereby create a behavior in which the robot will be able to track an object in the world frame and move towards it. While performing this behavior, the application would automatically subsume the lower level behaviors as they have already been implemented in these higher-level behaviors. Finally, some of these complex behaviors maybe further combined to create the intelligent behaviors which may be the Follow-Me module in our case. Thus, in this manner, the NAO robot can be programmed to interact with the environment intelligently. The next entity in question will be the various other connected devices of the owner (OD) which will play an important role towards making the NAO a more complete humanoid assistant. The various connected devices in the household of the owner will make the experience of having a humanoid assistant a more complete and fulfilling one. For example, the NAO humanoid can directly relay all the incoming notifications to the user through the network. Thus, the user can know about their current incoming calls, any new messages, email etc. from the robot itself without having to even use their cellphone. Another example of how connected living can make lives easy is through the use of networked ACs. Before leaving for work, the user can give the NAO instructions as to when he will return from work and then about 15 minutes prior to his arrival, the NAO can turn on the AC and thus when the user arrives at home, the temperature of his home is suitable and according to his desire.
Now to fully utilize the power of these ODs, the user must be able to trigger remote events on the NAO through his devices. These devices may either directly trigger events on the NAO or may store the event on the GAS which may trigger them on the NAO at an appropriate time. Here the architecture on how the remote events can be triggered on the NAO is explained in the following figure.
Figure 4 The asynchronous polling based I/O architecture
The asynchronous remote event triggering on the NAO has been made possible using the polling based asynchronous I/O. The process can be explained as follows: Initially a listener has been programmed on the NAO through the application to listen for any incoming notifications. Whenever, any such event is triggered on either the GAS or on any of the ODs, they send a pre-defined packet to the NAO’s listener socket containing the unique application id (AID) for which they want the NAO to react accordingly. Next, upon receiving the AID, if the NAO has been hard-wired to perform something (maybe say the estimated time to be taken to go from home to the user’s office), the NAO will perform that task straight forward. However, if the task needs some more details to be fetched from the GAS, then the NAO firstly authenticates itself to the GAS using its RID and AID and then when the GAS sends the required details of the task, the NAO then performs it. This extra layer of authentication ensures that even if any of the ODs get hacked by some intruder, the NAO won’t perform any such critical task without authenticating itself or the request received from the OD with the GAS. Also, through this architecture, the actuation by the NAO will remain confined by the set of applications defined on the GAS (all of which will carry a unique AID) and thus any such human-harming or intrusive tasks can’t be actuated through the NAO.
Finally, the most important entity in this application, the end-user won’t have much to perform other than use the robot as his companion. He will only have to register once using his Google® account through the web-based GUI of the GAS and fine-tune the services that he wants to use on the NAO. An example of this would be to choose the categories of the news that he wishes to listen to in the morning, or it may be to store the locations of his commute to and from his home so that NAO may tell him the accurate travel times through the various routes. Overall, personalizing his application would be totally left to the end-user and the application would provide him with multitudes of choices to choose from while installing the application in the robot.
[image:4.595.67.288.195.280.2]Figure 5 The demand-pool based IoT architecture
This proposed architecture will run the services on the GAS continuously and only when a robot asks for the specific service through a request supported by corresponding authentication via AID-RID request in the header, only then the GAS will send the corresponding details to execute the corresponding behavior on the robot. Hence, the behaviors will only run on the robot on a demand only basis and the robot has to pull the corresponding service from the pool of services available on the GAS. Thus, this architecture, in its very essence has implemented a SOA based model wherein multiple loosely coupled services give rise to a complete humanoid assistant through the NAO robot.
In details of the architecture it can be seen from Fig. 5 that the GAS will be composed of basically three different parts which will cater to the needs of the 3 different entities of our application. The first part is the web-server which will be responsible for accepting and storing the EU’s customization of various services offered through the application. It will also be responsible for displaying the missed notifications if the robot was offline when that notification was triggered. The second part of the GAS is the application which keeps track of all the robots currently active in the system and also the required tasks which will be running on those robots. It will also have the capacity to add more modules in the future as this part will be controlling the protocols for message passing between the various modules as well as the various external APIs. Basically, the Application Server (AS) will maintain a Model-View-Controller (MVC) architecture on the server side of the application. The third part of the GAS will be running a NAOqi broker, which is essentially an adapter program for the NAOqi OS which runs on the NAO robot. This program will get the actuation signals from the AS and will then interpret them in the forms of programs which will then run on the NAO robot. This layer also adds another extra layer of security because the control signals for the robot will not travel on the internet in form of plain TCP/IP packets, but will instead be Remote Procedure Calls (RPC) from the GAS NAOqi Broker to the NAOqi broker running on-board the actual the NAO robots. This broker-to-broker RPC interaction prevents the control signals from being modified while on transit by malicious hackers.
On the other hand, the NAO robot will simply run the native NAOqi broker for running the various behaviors which have been programmed as different NAOqi modules and this will be accompanied by a listener server, which will help the robot in detecting the remote events triggered by the GAS or
the ODs in an asynchronous manner. Hence, keeping in mind about these architectures in terms of the application and the network, the following section describes a particular use-case where some of the behaviors described in Section 3 were developed using the Choregraphe® SDK for the NAO robot.
V. USE-CASE DEVELOPMENT &ANALYSIS
To present a proof-of-concept as a part of this research, we have developed a prototype of this application. The modules which were implemented are as follows:
A. Read my email
This module has been developed using the POP email reader class in Python. This module basically fetches the authentication details of the end-user (EU) from the global application server (GAS) and then fetches email from the server and conveys them to the EU in an animated manner. B. Tell Traffic Situation
This module uses the DistanceMatrix API from Google to tell the real-time traffic information between the user’s office address and his current home address. This will help the user to get a heads-up before actually hitting the road.
C. Fetch Weather Information
This module uses the PYWAPI (Python Weather API) from Google to tell the real-time weather information for the user’s current location. The location is again traced via IP address of the EU’s current location.
D. Tell Story to Children
This module loads a story from the GAS according to the vice command from the EU and then tells the story in an animated manner to its audience. This module can also be used for entertaining guests by cracking jokes, making animated gestures etc.
E. Fetch Live News Updates
This module fetches the news updates from the GAS according to the categories defined by the EU. It uses all the popular RSS news source feeds like the CNN, the BBC etc. The news has been categorized into different categories like Headlines, Finance, Sport, Entertainment & Lifestyle, and Technology etc.
All of these modules had been developed and then they were inter-related using the Subsumption architecture which has already been discussed earlier. Then each of these modules were installed on the NAO as different services and then the GAS could turn them on or off depending upon the voice command of the EU or some remote triggering of events (like alarms) by the GAS. The results of this application have been discussed in the following section.
VI. RESULTS AND ANALYSIS
quite late in sending the results to the NAO. This resulted in an unwanted lag in the conversation the robot was having with the user. Also, the PYWAPI used for fetching the weather information sometimes produced time-out error, thereby throwing an exception in the Fetch Weather behavior. A comprehensive analysis of the various APIs used are given below.
API Name Description Critical Review MailParser A POP based
email parser class developed by Mark Lutz
It worked flawlessly every time. This class could be integrated with Gmail, Hotmail and other leading commercial email providers
DistanceMatrix API from Google for telling real-time traffic information
between a
source-destination pair
This also worked without any error. However, the only tricky part was to specify the address in a particular format to the API
PYWAPI Python weather
API by Google Although this API performed satisfactorily, yet it produced a time-out error sometimes, thereby raising an exception in the application
WolframAlpha The contextual web-search API by Wolfram Alpha
This API produced
really great
contextual web search results which were used to create an informative,
intelligent
[image:6.595.70.302.194.623.2]conversation with the end-user, however, the bandwidth of the robot’s connection must be high enough to prevent lag in the conversation
Table 2 Comprehensive review of the various APIs used
VII. CONCLUSION AND DISCUSSION
Building a complete humanoid assistant is a continuous process that will require substantial research for achieving perfectness. However, the present research has tried to develop an architecture which is scalable, robust, user friendly and easily expandable. It has utilized the existing elements in the IoT framework and has borrowed some existing elements from the TCP/IP suite. In this way, we have presented a service oriented architecture based application which is fully expandable without modifying the existing modules. This approach of creating plug-and-play based modules which could be added upon like present day android marketplace could only help the cause for making this
application feature-rich. Also, by specifying the priorities of these modules or behaviors, they could help the robot be more intelligent through the use of the reactive architecture. Thus, this research presents an architectural outline (with a number of behaviors as a proof-of-concept) through which the household acceptability of humanoid robots will be a reality in the near future.
ACKNOWLEDGMENT
The authors would like to express their heart-felt gratitude to the Robotics & AI Laboratory, IIIT Allahabad for providing the necessary infrastructure in the form of three NAO humanoid robots for implementing this research on them. Without these robots, this research would have only been a theoretical idea. Gratitude is also expressed to Dr. Pavan Chakraborty and Dr. Rahul Kala of the same laboratory for guiding and assisting the authors in the various stages of the research including the idea discussion, platform development etc.
REFERENCES
[1] Dautenhahn, K.; Woods, S.; Kaouri, C.; Walters, M.L.; Kheng Lee Koay; Werry, I., "What is a robot companion - friend, assistant or butler?," Intelligent Robots and Systems, 2005. (IROS 2005). 2005 IEEE/RSJ International Conference on, vol., no., pp.1192,1197, 2-6 Aug. 2005
[2] Dautenhahn, K., "Robots we like to live with?! - a developmental perspective on a personalized, life-long robot companion," Robot and Human Interactive Communication, 2004. ROMAN 2004. 13th IEEE International Workshop on, vol., no., pp.17,22, 20-22 Sept. 2004
[3] "Who is NAO?", SoftBank Robotics, 2017. [Online]. Available: https://www.ald.softbankrobotics.com/en/cool-robots/nao. [Accessed: 24- Mar- 2017].
[4] "BBC NEWS | Technology | Robots set to get homely by 2007",
News.bbc.co.uk, 2017. [Online]. Available:
http://news.bbc.co.uk/2/hi/technology/3764142.stm. [Accessed: 24- Mar- 2017].
[5] Robins, Ben, et al. "Robotic assistants in therapy and education of children with autism: can a small humanoid robot help encourage social interaction skills?." Universal Access in the Information Society 4.2 (2005): 105-120.
[6] Robins, Ben, Kerstin Dautenhahn, and Paul Dickerson. "From isolation to communication: a case study evaluation of robot assisted play for children with autism with a minimally expressive humanoid robot." Advances in Computer-Human Interactions, 2009. ACHI'09. Second International Conferences on. IEEE, 2009.H. Poor, An
Introduction to Signal Detection and Estimation. New York:
Springer-Verlag, 1985, ch. 4.
[7] Autism NAo study: Shamsuddin, S.; Yussof, H.; Ismail, L.; Hanapiah, F.A.; Mohamed, S.; Piah, H.A.; Ismarrubie Zahari, N., "Initial response of autistic children in human-robot interaction therapy with humanoid robot NAO," Signal Processing and its Applications (CSPA), 2012 IEEE 8th International Colloquium on , vol., no., pp.188,193, 23-25 March 2012
[8] Tapus, Adriana, et al. "Children with autism social engagement in interaction with Nao, an imitative robot–A series of single case experiments." Interaction studies 13.3 (2012): 315-347.
[9] Chapter 1: Service Oriented Architecture (SOA). Msdn.microsoft.com. Retrieved on May 15, 2015.
[10]Brooks, R.A., "A robust layered control system for a mobile robot," Robotics and Automation, IEEE Journal of , vol.2, no.1, pp.14,23, Mar 1986
