Applications and deployment

State of the art and current research

In recent years, there have been a number of longterm real-world demonstrations of plan-based con-trol, which have impressively shown the potential impact of this technology for future applications of autonomous service robots. In NASA’s Deep Space program a plan-based robot controller, called the

Remote Agent, has autonomously controlled the performance of a scientific experiment in space. In the Martha project, fleets of robots have been effectively controlled and coordinated. Xavier, an au-tonomous mobile robot with a plan-based controller, has navigated through an office environment for more than a year, allowing people to issue navigation commands and monitor their execution via the Internet. In 1998, Minerva, another plan-based robot controller, acted for thirteen days as a museum tour guide in the Smithonian Museum, and led several thousand people through the exhibition.

Plan-based control of autonomous helicopters for traffic surveillance and assistance In the WITAS project [DGK⁺00] the control software of a small, unmanned helicopter carrying computers, video cameras, and other electronic equipment is developed. The task of the helicopter is to observe what goes on on the ground, in particular traffic situations on roads, and to make decisions on that basis. It is therefore required to “understand” what happens on those roads – conventional maneuvers of individual cars and other road vehicles, dangerous or otherwise exceptional maneuvers, structure of the traffic e.g. congestion. It is also required to perform tasks that are assigned by the operator or triggered by the observations it makes itself, for example to follow a certain car that flees from the scene of an apparent crime, or to assist a certain car so that it can make it through difficult traffic and get to a particular destination as quickly as possible, or to deliver a particular parcel to a particular point. The helicopter is instructed by a human in very abstract terms. The most important capabilities for the WITAS system are therefore (1) to form a model of (“understand”) scenes and events that it observes on the ground, and (2) to predict, plan, and make decisions using that model.

Plan-based control of tour guide robots [BAB⁺01] implemented a plan-based high-level con-troller for an interactive museum tour guide robot. The tour guide robot, called MINERVA, has operated for a period of thirteen days in the Smithsonian’s National Museum of American His-tory [TBB⁺99]. In this period, it has been in service for more than ninety four hours, completed 620 tours, showed 2668 exhibits, and traveled over a distance of more than forty four kilometers. The plan-based controller controlled MINERVA’s course of action in a feedback loop that was carried out more than three times a second. MINERVA used plan revision policies for the installment of new commands, the deletion of completed plans, and tour scheduling. MINERVAperformed about 3200 plan adaptations. The MINERVAexperiment demonstrates that plan-based controllers can (1) reli-ably control an autonomous robot over extended periods of time and (2) relireli-ably revise plans during their execution.

Plan-based robot control for transshipment tasks The Martha project [AFH⁺98] has investi-gated the management of a fleet of autonomous mobile robots for transshipment tasks in harbors, airports and marshaling yards. In such contexts, the dynamics of the environment, the impossibility to correctly estimate the duration of actions (the robots may be slowed down due to obstacle avoid-ance or re-localization actions, and delays in load and unload operations, and so on) prevent a central system from elaborating long or medium term efficient and reliable detailed robot plans.

In the Martha project, a more flexible way is pursued in which the robots to determine incrementally the resources they need taking into account the execution context. As a consequence, [AIQ98] have developed advanced techniques for plan merging in order to improve multi robot cooperation.

Plan-based control for autonomous spacecrafts The Remote Agent is a reusable control system that enables goal-based spacecraft commanding and robust fault recovery [MNPW98a]. The Remote Agent accepts high level goals from the operators and on-board software modules such as the au-tonomous navigator. The Remote Agent determines a plan of action that will achieve those goals and carries out that plan by issuing commands to the spacecraft. The Remote Agent considers the current state of the spacecraft both in planning how to achieve its goals and in executing that plan. This allows for robust responses to failures and other contingencies. The Remote Agent detects and re-sponds to failures in real-time and if necessary generates a new plan for achieve its remaining goals.

The REMOTEAGENT has successfully operated the Deep Space 1 spacecraft for two days in May of 1999. This experiment was the first time that an autonomous agent has controlled a deep space mission.

Other plan-based robot controllers Other plan-based robot controllers include XAVIER, an au-tonomous mobile robot that has performed longterm navigation experiments in an office build-ing [SGH⁺97a]. XAVIER has been in nearly daily use for more than one year and traveled in this period more than 75 kilometers in order to satisfy over 1800 navigation requests that were specified using a World Wide Web interface. [FPS⁺96] have developed CHIP, an autonomous mobile robot that is intended to serve as a general-purpose robotic assistant. The main application demonstration of CHIP was the Office Cleanup event of the 1995 Robot Competition and Exhibition. CHIP was to scan an entire area systematically and, as collectible objects were identified, pick them up and deposit them in the nearest appropriate receptacle. Their work is very interesting because they have developed a library of low-level plans that can be carried out in situation-specific ways.

Research goals and challenge application scenarios

The control of autonomous service robots can be made more challenging along several dimensions:

the capabilities of the robot, the nature of the environment, and the complexity of the tasks to be performed.

In this section we present a selection of challenge application scenarios that have been developed during the second seminar “Plan-based Control of Robotic Agents”. The scenarios are characterized by varying degrees of difficulty along these dimensions. The different application scenarios are:

autonomous robots with sophisticated manipulation skills, the robot companion, autonomous space-craft control, plan-based control in intelligent, sensor-equipped environments, and the autonomous household robot.

Autonomous robots with sophisticated manipulation skills Our first challenge application sce-nario is to have a robot or a team of robots that are mobile and equipped with arms and hands.

The robots are to assemble either pieces of Ikea furniture or assemble specified models using toy construction sets such as Lego, Fischer Technik, Baufix, etc. Thus, the objective is the develop-ment of mobile robots with manipulators that are taskable in natural language and are teachable by demonstration.

The reason that we propose this challenge scenario is that it exemplifies a number of the hard prob-lems in robot planning and plan-based robot control. First, adding manipulation will increase the

complexity of the tasks the robots can do, and so will require more high-level planning and learning than what is currently needed. Second, communication in natural language requires advances in user interaction and the sensor-based grounding of symbolic descriptions in visual scenes. In particular, the robot needs to disambiguate between sensed objects that might satisfy a given symbolic descrip-tion. Also, planning mechanisms need to be extended to deal with symbolic object descriptions that might be incorrect, inaccurate, incomplete, or ambiguous. Another research challenge is the appli-cation of planning techniques for speeding up learning by demonstration and/or being told. A fourth issue is that such open assembly tasks require new planning mechanisms that properly integrate sym-bolic task planning with motion, grasp, and roadmap planning. Finally, the robots need much more realistic models of construction plans and designs of pieces and what can be done with them as it is typical for many of the recent planning application tasks.

In the case where the robots are to assemble furniture from instructions additional challenges such as the manipulation of large objects and the cooperative manipulation with multiple robots are posed.

In addition, the tasks have to be dynamically distributed among the robots in the team.

Robot companion Our second application challenge scenario is the development of a robot that serve elderly people as a companion in their daily life. Such robot companions must be able to evolve and grow their capacities in close interaction with humans in an open-ended fashion. Quite intuitively the notion of companion includes a variety of issues. The robot must provide continual operation over long periods of time; provide sufficient coverage and monitoring without being intru-sive; maintain the privacy of the users; interact with users and guarantee their safety. The robot is not only considered as a ready-made device but as an artificial creature, which improves its capabilities in a continuous process of acquiring new knowledge and skills. Besides the necessary functions for sensing, moving and acting, such a robot will exhibit the cognitive capacities enabling it to focus its attention, to understand the spatial and dynamic structure of its environment and to interact with it, to exhibit a social behavior and communicate with other agents and with humans at the appropriate level of abstraction according to the context.

From the point of view of plan-based robot control a robot companion needs planning mechanisms that are very different from those needed for construction set assembly. The companion challenge emphasizes mechanisms for integrating additional plans into existing commitments, elaborating and expanding them as needed, and monitoring their execution.

Besides the technological challenges the application challenge scenario also holds the potential for making very substantial contributions to the ever more important societal problem of aging popula-tions. We believe that because of privacy considerations autonomous robots are expected to achieve better acceptance than intelligent sensor-equipped rooms.

A variant of the robot companion for elderly people is the deployment of plan-based robots that can navigate in a crowd in large environments (Disneyland, large shopping malls), take people who are lost to their destination, guide tours, carry suitcases, etc. Technical challenges posed by this scenario include the navigation and mapping of large areas requires advances in mapping, cooperation of multiple robots and sensors; the navigation in a crowd requires fast vision processing, detecting people intentions, using social conventions, etc.; and finding people who need help requires detecting user distress level, interacting with them, and adapting to their reactions.

Spacecraft control Plan-based robot control already plays an important role in NASA’s research agenda for the exploration of space, the investigation of dangerous places on earth, including vol-canos, and the probing of other planets and orbs. The deployment of intelligent autonomous robots and spacecraft enables NASA to perform more science missions at lower cost, perform science ex-periments that are much deeper in space, to take advantage of unexpected science opportunities, and run unmanned spacecraft missions of longer durations. The autonomy and plan-based control techniques therefore allow for running a greater variety of missions at lower cost.

Planned NASA missions include sending teams of planetary rovers to the Mars in order to explore the surface more thoroughly, orbiter missions around the Jupiter moon Europa, and various deep space missions. The Jupiter moon Europa is believed by a number of astrobiologists to provide conditions that might enable the evolution of a biosphere. Unfortunately this biosphere lies in a hypothesized ocean under a very thick crust of ice. Eventually astrobiologists intend to send an autonomous hydro-bot to Europa that drills itself through the ice, explores the ocean, and sends back the results from the scientific experiments. Obviously, such a mission would require sophisticated planning capabilities on the robot.

Other example applications of robot planning include: spacecraft commanding and payload op-erations; planning and scheduling for process control; planning and scheduling for robotic space activities; operations of air, space and ground-based scientific observatories; scheduling of critical resources whether on the ground or on-board; science data analysis; design and analysis of space-craft systems; planning and scheduling of scientific experiments; and planning and scheduling of crew activities.

Space applications of robot planning entail a number of important research questions. First, they imply extremely high demands on reliability, safety, and robustness. They also have require sophisti-cated task planning and scheduling techniques, true autonomy, and reasoning about detailed models of the controlled system.

Household robot challenge Realizing an autonomous household robot with sophisticated ma-nipulation skills is an important longterm challenge application for the domain of plan-based robot control. In the last Dagstuhl seminar on “Plan-based Control of Robotic Agents” it was selected as a hallmark problem for guiding the further development of plan-based robot control techniques.

We consider a humanoid robot, such as the Sony SDR-3 or the Honda Asimov, with additional manipulation skills that is to do household chores as an interesting challenge for the field of plan-based control of autonomous robots. The challenge is to develop a plan-plan-based controller for such a robot that enables the robot to be put in another household, to operate in this household for some months, and do a substantial part of the household chores satisfactorily. To meet this challenge, the robot must acquire models of its environments not only a map of the apartment but also models of the daily rhythm of the household, the time the dishwasher takes to clean the dishes, etc. and it must use these models to better manage its activities. It should also acquire or generate plans to perform its tasks, such as cleaning the living room and it should build up models of these activities that include information such as how long it typically takes to clean up the rooms. The robot is to do several things at a time for example, cleaning while baking a cake. These activities have to be interruptible:

the phone might ring in the midst of cooking. Many of the activities require interactions between the robot and the environment and considering actions, such as cleaning the kitchen, to be discrete

does not suffice in many situations. These are only some of the aspects that require much more sophistication in controlling such robots than is provided by current robot planning mechanisms.

Develop a robot that acquires knowledge from ontologies.

Other scenarios Other interesting application challenge scenarios include but are not limited to RoboCup Rescue, logistics scenarios, intelligent sensor-equipped living environments, autonomous robot soccer, and plan-based driver assistance systems.

The goal in RoboCup Rescue is to provide emergency decision support by integration of disaster information, prediction, planning, and human interface. A generic urban disaster simulation envi-ronment is provided. Heterogeneous intelligent agents such as fire fighters, commanders, victims, volunteers, etc. conduct search and rescue activities in this virtual disaster world. Real-world inter-faces such as helicopter image synchronizes the virtuality and the reality by sensing data.

Thus, RoboCup Rescue studies disaster rescue after catastrophes like earth quakes as a testbed for research in controlling very large numbers of heterogeneous agents in hostile environments. Re-search issues include multi-agent team work coordination, physical robotic agents for Re-search and rescue, information infrastructures, decision-theoretic planning techniques for decision support sys-tems, rescue strategies and robotic systems that are all integrated into a comprehensive systems in future. Interesting planning challenges include multi-agent planning, real time/anytime planning, heterogeneity of agents, robust planning, mixed-initiative planning.

Another interesting scenario is the control and the management of a fleet of autonomous mobile robots for transshipment tasks in harbors, airports and marshaling yards and supply chain monitor-ing and control in general. Here, plan-based robot control and schedulmonitor-ing techniques can plan and execute supply chains that are faster, and can react more spontaneous to problems.

Autonomous robot soccer is another challenge domain. Here, the dynamics of the game situations prevent the application of sophisticated online plan generation mechanisms. Yet, plans, plan-based control, and playbook learning seem to be important for acquiring sophisticated playing skills. A last domain that we would like to mention here is the enhancement of intelligent driver assistants systems through plan-based control mechanisms.