System in application

The last test to be presented here includes some scenes from a shopping experiment – involving all available functionalities – where the robot is hindered in executing given tasks by moving obstacles (e.g. other shop- pers with shopping carts). Fig. 5.19 shows a single shopping run in a small laboratory supermarket setting. The same experiment will be presented as well in the next two chapter focusing on the user interaction and the multi-robot coordination.

Fig. 5.18.: Photos showing some of the performed tests using a ordinary shopping cart and InBOT’s on-board object tracker. Others have been done using ETrolley to gain precise and robust localization information on the moving object.

The user is asked to pick up five products while other shoppers are moving around him with their carts. First the robot guides the user to the first product and simultaneously avoids the moving obstacle (A), later while operating in the Following Mode the robot keeps a safe distance from obstacles (B) and (C). In all cases the robot did not move into other objects and did not collide with its user (as described in Chapter 3 the presented behaviors are merged with the behaviors for handling static obstacles and the safety module counterchecks all movement commands in the end).

5.4. Discussion

Some methods for avoiding moving objects proposed in the literature only steer the velocity of the robot on the path planned around the static obstacles, which is only sufficient for moving obstacles crossing the path, not for ones moving along the same or a similar path. Others are directly coupled to the path finding algorithm. This is not sufficient as well because in a dynamic environment the robot can also be hit by moving objects while standing still. In general, the advantage of the deliberative planners compared to the reactive methods is obvious – especially when state-of-the-art hardware renders the time delay for the computation insignificant. But the scenario of this thesis lacks too much information (limited field of view, no exact and reliable map, sometimes no target location is available e.g. in Visual Servoing Mode or Manual

Steering Mode) to be able to rely always on a planner-based approach only.

Therefore, in this thesis a combination of a planner with reactive behaviors is used. Again a three-leveled approach is proposed: First, a reflex moves the robot directly away from mobile obstacles which came too close or suddenly started moving, enabling the robot to regain a safety distance. This reflex is kept extremely simple, this way it is able to dodge moving objects even before a prediction of the movement direction can be made. Second, a reactive behavior lets the robot move out of the predicted path of an approaching object. Third, to solve complex situations, the behavior-based components are topped by a spatio-temporal planner which generates a safe and efficient long-term path.

5.4. Discussion

Fig. 5.19.: Final test for avoiding moving objects – a shopping run involving all available functionalities: the user (tracked by the blue line) starts together with the robot (grey rectangle, red line) in the lower right corner. The robot guides the user to the first product (1) while avoiding the moving shopping cart (A) (indicated by the yellow dotted line): the strong bend to the left at the yellow flash symbol. After following the user to the next products (2) and (3), the robot continues to product (4) and avoids the crossing cart (B) (orange rotted line) by slowing down and letting it pass. The robot continues by aligning itself behind the cart (orange flashes). While following to the final product (5) the robot is cut by cart (C) (red dotted line). Again, the robot lets the cart pass (red flash). (Robot’s path red, critical areas: flash symbols, the user – who is continuously impairing the robot being another close-by moving obstacle – is tracked by environmental cameras resulting in the chaotic blue line. The bad quality of the user’s track comes from continuous occlusions by other objects (shelves, robot, carts, other shoppers, etc).

In several tests the concept has proven its performance. The system was designed to avoid accidental col- lisions and performs well under this assumption. If someone should really try to hit the robot he will succeed due to the velocity and acceleration limitations applied on a robot operating in a populated environment.

A crucial challenge in avoiding moving objects is the robust and timely detection and tracking of the objects. In this thesis two components were utilized to provide the necessary data: First this were two planar laser scanners mounted on the robot at feet height. This is not sufficient for robustly tracking persons as occlusions are frequent and the field of view is very limited. Additionally tracking a group of walking persons is prone to errors because the persons lift the feet above the measurement plane of the sensor when walking. Therefore the second component used two planar laser range finders mounted in the environment at chest height whose data has been merged with the data from environmental cameras. The drawback on this system has been the additional time delay resulting from the transfer of the data to the robot using WiFi.

6. User Interaction

After the navigation system has been described, this chapter approaches the second major challenge: the human-robot interaction (HRI) capabilities of the control system. A special focus is on a challenge common for cooperative actions of robot and user: sharing the control between robot and user smoothly. This way the robot can assist the user with the implemented abilities without putting the user out of control. Preluding, the interactive capabilities, modalities for the users’ input, and the modes of operation of InBOT will be discussed.

As the shopping assistance robot shall serve its user, interaction with the user is an important topic. Having a powerful navigation system alone may serve the robot well, but not the user. The user needs to have access to the functionalities in an intuitive way – he wants to do shopping and not to control a robot – and the robot must be able to fulfill tasks in cooperation with the user. Especially handicapped users could benefit from the provided assistance functionalities: visually impaired customers can command the robot by speech and let themselves be guided around obstacles and directly to the desired products. Weak or elderly people do not have to push the robot themselves. But especially when cooperating with these groups of people an robust HRI system is even more important.

Fig. 6.1.: InBOT is guiding a user Fig. 6.2.: InBOT is controlled by the force sensitive

handle

In contrast to tele-operated or autonomous robots, in the setting of this thesis the user is always present, even impairing the robot’s motion. Thus, this chapter shall in particular focus on a challenge common for

Fig. 6.3.: Inserting control data into the architecture using interfaces between layers. Insertion of user input between the safety behaviors and the Hardware Abstraction Layer is not allowed to ensure that the safety behaviors are always in charge.

all cooperative actions of robot and user: the trading and smoothly sharing of control. As baseline for the interaction the modalities available to the user for controlling the robot are identified. They inform the robot about what the user actually wants to do. A second foundation for the interaction are the modes of operation (two examples can be seen in Fig. 6.1 and Fig. 6.2) which define the current setting for the cooperative task execution. The chapter will be concluded by showing the application of the concept of control sharing and

Trading in some experiments with the robot InBOT and human users.

As discussed in Chapter 3 “The Hybrid Control Architecture”, the concept of control sharing is not implemented by associated behaviors. It emerges from a multitude of components and their interplay as defined by the architecture, such as but not limited to the integrated multimodal communication layers, the force sensitive handle, the user tracking, or finally the adaptive behaviors. The HRI system spans the complete control architecture and thus is present on all levels of abstraction. Very important for control

sharing is the control data flow as it defines how the user can influence the robot’s behavior. The control

architecture defines user-input orthogonally to the usual control data flow by providing interfaces and fusion behaviors on top of each of the layers (see again the sketch shown in Fig. 6.3).

Scope of this chapter: Due to the enormous wide range of the field of HRI, the content implemented in the course of this thesis will be limited to the components and concepts relevant for the application of shopping assistance in the supermarket, for example implementing the guiding and following behaviors, giving orders to the robot, receiving feedback from the robot, and sharing the control during task execution. The HRI shall focus on the human acting as the customer to whom the robot is attached, other humans are regarded only as moving obstacles in most of the cases. Humans acting as operators or supermarket staff will be omitted here completely. They can either control the robot using the same modalities as the customers do or exert direct control using the MCAGUI. Further concepts like interpreting the humans’

6.1. Five modes of operation