Task Directed Programming of Sensor Based Robots

Task Directed Programming of Sensor Based Robots

B. Brunner, K. Arbter, G. Hirzinger

German Aerospace Research Establishment (DLR), Oberpfaffenhofen, Institute for Robotics and System Dynamics,

Postfach 1116 D-82230 Wessling, Germany email: [email protected]

Abstract

We propose the so-called TeleSensorProgramming con-cept that uses sensory percon-ception to achieve local au-tonomy in robotic manipulation. Sensor based robot tasks are used to define elemental moves within a high level programming environment. This approach is ap-plicable in both, the real robot’s world and the simu-lated one. Beside the graphical off-line programming concept, the range of application lies especially in the field of teleoperation with large time delays. A shared autonomy concept is proposed that distributes intelli-gence between man and machine. The feasibility of graphically simulating the robot within its environment is extended by emulating different sensor functions like distance, force-torque and vision sensors to achieve a correct copy of the real system behaviour as far as possible. These simulation features are embedded in a task driven high level robot programming approach. This implicit programming paradigm is supported by a sophisticated graphical man machine interface. Pro-gramming by demonstration is performed by introduc-ing an analytical controller generation approach. Sen-sor fusion aspects with respect to autonomous senSen-sor controlled task execution are discussed as well as the interaction between the real and the simulated system.

1

Introduction

Programming a robot off-line has been restricted to showing a sequence of cartesian positions or joint angle configurations. But mechanical inaccuracies at the manipulator level and dif-ferences between real and simulated world lead to difficult problems in the execution of off-line defined robot activities. Local adaptations on path generation level have to be done via teach-in commands by human operator, especially in the context of manipulation tasks, such as gripping or handling objects. Even small deviations in position, orientation and shape of all the objects to be handled are not allowed, be-cause the task execution will fail in an uncertain environment.

Graphical simulation of robot tasks and downloading the gen-erated commands to the real robot is limited to the joint or cartesian motion level. This approach is only useful if geo-metrical consistency of the real environment and the simulated one can be guaranteed. This is a demand that cannot be met with available programming systems.

The most important requirement to achieve autonomy is the ability to successfully execute a given task by intelligent sensor data processing. Actual sensory data are matched with a predefined reference pattern to get information for the robot controller to achieve the desired end position. Local deviations from the desired state are detected by the sensors and handled by the robot system in an autonomous way. Therefore sensor controlled movements have to be used to bring local autonomy onto the machine level. High level planning facilities for task scheduling as well as intelligent error handling mechanisms are required for full autonomy but state-of-the-art techniques are insufficient to provide the adequate tools. Presently full autonomy is not reachable.

2

The TeleSensorProgramming approach

Therefore we favour a shared autonomy concept that dis-tributes intelligence between man and machine [1]. Presum-ing that sufficient information about the actual environment is available from the sensor systems, partial tasks can be exe-cuted independently on the machine level. Specifications and decisions on a high task planning level have to be done by human operators. Local sensory feedback loops are executed by the robot system, while global task level jobs have to be specified interactively by a human operator. Coarse planning activities have to be done on a task oriented level by human in-telligence, fine path planning on manipulator level takes place on a sensor based control level [2].

This shared autonomy approach is the basis of our program-ming paradigm for that we have coined the term

TeleSensor-Programming (TSP). This means a new way for robot

pro-gramming on a task directed level. The teach-in of a robot system occurs not on the joint or cartesian manipulator level

but on a high language level, i.e. the operator plans activities on a level which can be worked off by the robotic system independently from human intervention. Hence TSP means teaching by showing typical situations on a task directed level including nominal sensory patterns with the aid of sensory re-finement in the real or a completely simulated world. All the taught sensor based actions can be activated in analogous situations. virtual feedback real feedback Graphics and Animation Images Plots Graphics

controller robot_model sensor_models environment _model

controller robot sensors environment local autonomy loop

local autonomy loop

task planning and error handling path planning and parameter determination

world model data base sensor fusion algorithms a posteriori knowledge

update of world model a priori knowledgeusage of world model

(Ground) Simulation (Remote) Real System real system control loop predictive control loop

Figure 1 The TeleSensorProgramming Concept

Figure 1 shows the structure of our TSP concept, consisting of two parallel working control cascades. One of them is the real system control loop, containing internal control loops for local autonomy. The other one is a simulation environment which is structural equivalent to the real system, with few exceptions. The most important is a signal delay which may occur in the real (remote) system, e.g. in space applications, which is not implemented in the simulation. This makes the simulation predictive with respect to the real system. Another exception is the display of internal variables in the simulation part, which cannot be observed (measured) in the real system. This gives the operator or task planner more insight to what happens or may happen on the system according to his commands. Communication between the two loops runs via a common model data base which delivers a priori knowledge for the execution on the remote level and a posteriori knowledge for the model update of the simulated world.

For such an intelligent programming system different tools are necessary to implement the required functionality. First a sophisticated simulation system has to be provided to emu-late the real robot system. This includes the simulation of sensory perception within the real environment. Beside this sensor simulation, the shared autonomy concept has to provide an efficient operator interface to setup task descriptions, to configure the task control parameters, to decide what kind of sensors and control algorithms should be used, and to debug an entire job execution phase.

Two applications in the field of robotics are evident to the proposed TSP approach. First the graphical off-line

program-ming concept is extended by the processing of simulated

sen-sory data and by the introduction of local autonomy on the task description level. This means that not only the joint and cartesian information is gathered by graphically guiding the robot during the off-line programming task, but also the simulated sensory informations are stored off-line as nominal patterns for subtask execution on a local feedback loop. The fine motion control loop for handling uncertainties takes place independently of any human intervention both at the simula-tion side and the real one. A main advantage of this off-line programming scheme is the feasibility to verify all the robot actions before real execution is performed. This includes the perception and processing of sensory data and the activation of the sensor controlled elemental moves.

Second, the field of telerobotics with large time delays espe-cially in space and subsea applications, is a good area for this sensor based, task directed programming approach [1]. Di-rect visual feedback in the control loop, where a time delay of a few seconds is inherent, is not feasible for the human operator to handle the robot movements in a suitable way. Predictive simulation is the appropriate means for an opera-tor to telemanipulate the remote system on-line [3]. Similar approaches are known, which make usage of force reflecting hand controllers to feed back force sensor signals for shared and teleoperated control modes [4]. Furthermore an interac-tive supervisory user interface makes it possible to set up the environmental and control parameters. But in our approach, the sensor based elemental move concept, feasible for all kind of sensors and execution tasks, are to the fore. The operator has only to guide the robot through the task space in a coarse way and to activate specific sensor control phases. Sending these gross commands to the remote system enables the real robot to execute the desired task with its own local auto-nomy after the delay time has elapsed. The main feature of our telerobotic concept is to replace the time-delayed visual feedback by predictive stereo graphics with sensor simulation, providing a supervisory control technique, that will allow to shift more and more autonomy and intelligence to the robot system.

The main focus of this paper lies in the task-directed pro-gramming approach which facilitates the robot propro-gramming work by an intuitive task description level. Predefined com-plex sensor based tasks can be described on a task driven level which allows for usage in a varying context. To de-scribe robot activities on a high language level we introduce the elemental move concept. This concept allows us to define subtasks that can be executed by the robot system in an au-tonomous way. To program a complex robot task it is only necessary to define a sequence of elemental moves on an in-tuitive planning level. An analytical approach is proposed to define sensor based elemental moves without an extensive

controller design. Therefore we call our method

program-ming by demonstration, which represents an easy variant of

teaching by showing.

3

Elemental move concept

In the past many attempts have been started to describe robot assembly plans on a high language level. Command language approaches [5] or planning tools such as Petri nets [6] or assembly graphs [7] have been proposed. They work well in a structured, well-defined environment for a specific application field like block world assembly [8] or compliant motion tasks [9].

Our task directed programming approach is driven by an elemental move concept, that enables us to program robot tasks in an intuitive implicit manner. A complex robot task is composed of such elemental moves (EMs) that can be divided into three categories. First pose controlled EMs that are described by the goal pose (position and orientation) in the cartesian space, second the sensor controlled EMs that have to be defined via nominal sensory patterns, and third shared controlled EMs. Each class of EMs has a template of pre-and postconditions that describe the prerequisites to activate or stop the execution of an EM instance. An EM instance can be regarded as a node of a state machine. The transition between two nodes is defined by the matching of post- and preconditions of the corresponding EM nodes.

3.1 Pose controlled elemental moves

Pose controlled EMs are defined via the goal pose in the cartesian space. Therefore the precondition template for the pose controlled EM class is empty, the postcondition is met if the tool frame of the robot has reached the desired pose within the predefined limits. The path generation considers the inverse kinematic problem that is solved by an iterative approach. Currently an on-line collision avoidance algorithm is implemented to find a collision free path from the current start to the defined end pose [10] .

3.2 Sensor controlled elemental moves

Sensor controlled EMs are defined via the nominal sensory pattern in the reference pose relative to the environment. Therefore the precondition template for the sensor controlled EM class includes the availability and correctness of the sensory data, which is necessary to apply the control algorithm for achieving the desired goal state. This class of EMs represents full autonomy, in the sense that the sensor based control mechanism leads to a properly system behaviour. An EM is fully sensor controlled, if all 6 degrees of freedoms (DOFs) in the cartesian space are controlled by sensor values.

With the aid of sensor based path refinement it is possible to act in an environment under the constraints of the sensor measurements. This class of the sensor controlled EMs also gives us the possibility to handle similar tasks in analogous situations. E.g. Approach moves, that will be able to align the robot gripper with the object to be picked up, can be defined as one sensor controlled EM for a desired distance and/or vision sensory pattern. Similar objects at different poses can be gripped by the same EM.

3.3 Shared controlled elemental moves

The shared controlled EM class is a mixture of the previous two. For each instance of shared controlled EM the constraint space configuration has to be defined. The constraint space configuration description specifies which degree of freedom in the cartesian 3D-space is pose or sensor controlled. For instance a contour following task in the xy-plane can be described by the pose controlled DOFs ztrans, xrot, yrot, zrot

and the sensor controlled xtrans and ytrans. Based on Mason’s

C-frame concept [11] sensor controlled subspaces are defined using nominal sensory patterns to control the selected DOFs. The free DOFs are controlled by the path planning algorithm or in the case of teleoperation by the human operator using a 6–DOF input device. The techniques used for projecting gross move commands into the pose and sensor controlled subspaces have been discussed in a number of previous papers, e.g. [12].

4

High level task directed programming

Task directed robot programming has been discussed so far in the field of generalized planning [13] and assembly plan generation as an AI paradigm [14],[15]. Especially for the telerobotic applications interactive planning facilities are of crucial importance. To integrate such planning tools into the framework of sensor based task execution, only theoretical work has been done so far [16]. We believe that the real-ization of task directed robot programming by the elemental move approach leads to an intuitive high level programming concept.

Complex robot tasks are described by an action graph of ele-mental moves. The nodes of this graph are represented by the EM controllers. The transitions between the corresponding nodes are defined by the matching of the successor precon-ditions with the predecessor postconprecon-ditions. A plausibility check has to be run to look for the congruence between the end conditions of the subtask and the start conditions of the following one, so that the consistency of the subtask sequence is ensured within the entire robot task. The feasibility of an EM, i.e. the question whether the preconditions can be met, depends on availability of the required sensor values as well as on determination of the constraint space configuration.

To implement a real task directed programming tool we pro-pose a hierarchical declaration of the EM framework. At the lowest level we define the pose, sensor, and shared controlled EMs as so-called atomic elemental moves that cannot be split into several control items. Each node in the action graph represents one instance of an atomic EM. A sequence of con-nected nodes as a part of the action graph can be joined to a so-called composed elemental move, which can be further used as a logical EM on a high task description level. We have to consider that the uniqueness of the graph transitions are guaranteed by the alignment of the pre- and postconditions of successive EMs. Furthermore a composed EM is put to-gether by a non-empty sequence of atomic and/or composed EMs in a recursive way.

One remark to the definition of the atomic EMs: In contrast to other approaches like [5] or [4] we don’t have to predefine all the elemental operations which can be used to execute any task. We can interactively define and configure an atomic EM via programming by demonstration which is the topic or the next chapter.

5

Programming by demonstration

Visual programming tools for the direct transfer of human ma-nipulation skills to the robot have been developed, but they are constrained to location gathering by using a marker-based vision system [17] or trajectory generation from observing object features [18]. Compliance controller identification by human demonstration [19] is restricted to straight line mo-tion in well-defined constraint configuramo-tions. Methods pro-posed during the last years [20],[21],[22] for programming by demonstration utilize the real robot system to collect the required demonstration data. The same system is used for further execution, so that this class of direct demonstration approaches can be regarded as simple playback methods.

5.1 Remote demonstration

We have extended this methodology to the separation of demonstration and execution system that gives us the ability to define task level robot actions off-line or to overcome the problems attended upon time-delayed telerobotic applications. Before the execution of the robot task in the real system takes place, we demonstrate the task by using sophisticated simula-tion tools including the emulasimula-tion of the interacsimula-tions between the manipulator and the environment, concerning collision in-formation of contact sensors as well as non-contact sensor devices. Currently we have implemented the functionality of laser distance sensors, a feature-based simulation of stereo cameras, and the behaviour of a stiff force-torque sensor in contact with the environment [23]. We emphasize that the usage of simulated sensory behaviour is only feasible, if the

virtual system is properly calibrated. Especially in the emula-tion of computer vision the knowledge of internal and external camera parameters is indispensable for finding the right map-ping between the 3D world model and the 2D camera planes.

5.2 Defining the atomic elemental moves

It should be noted that the following concepts for teaching atomic elemental moves are applicable to both, first to the simulation with regard to graphical off-line programming or telerobotics, second to the real robot (including sensor equip-ment) without any simulation environment.

Applying programming by demonstration to simple pose

con-trolled EMs is straightforward. The actual cartesian pose of

the robot’s tool frame is stored with current manipulator data like the velocity or acceleration. On activation of such a pose controlled EM the robot moves from any pose to the desired one regarding the inverse kinematic problems as well as the collision detection. To move the robot a 6DOF input device is used, which allows the operator an easy way to generate such target points within the robot’s workspace.

To define the various instances of the sensor or shared

con-trolled EM class we propose an intuitive programming by

demonstration methodology. The key issue is to find an ap-propriate controller to perform the desired EM correctly.

5.3 Automatic controller generation

For the handling of the sensor controlled EMs we outline a method to estimate the motion parameters by the known sensor values to achieve the predefined goal state of the robot’s pose (cartesian position in translation and orientation or joint angle position). In other words we have to find a control sequence that transduces the robot’s end effector from a pose x0 into the nominal pose x* which is described by its

corresponding sensor values y*_{. x}*_{is assumed to be unknown}

or uncertain. The motion parameters relative to the sensed environment are expressed by the vector increment xk, the

actual sensor values by the vector yk. The nominal sensor

values y* generally are non-linear functions of the actual interaction between the robot, the sensor and the environment in the robot’s nominal pose x*_.

(1) In the execution phase we want to find a controller sequence, that is able to reach the goal state dependent on the nominal sensor value vector, which the system was taught for. Starting from a pose x0 we calculate stepwise the motion parameters

xkin order of the actual and nominal sensor values, i.e. we

apply a linear controller of the form

where C is a constant N*M-Matrix, which maps the displace-ment in the M-dimensional sensor space to the N-dimensional control space, and where

is a scalar damping factor, which determines the dynamic behaviour of the closed control loop. The optimal controller coefficients, in the sense of least square estimate, are expressed by the pseudoinverse

(3) where the elements of J are the partial derivatives of the sensor values y to the motion parameters x:

(4) The teach-in process is implemented in an efficient way, because the nominal sensor values y* _{can be easily derived}

from either the simulation or the real system. The controller design, i.e. the construction of J is also implemented in a simple manner by experimentally estimating the Jacobian J, i.e. moving the simulated or real robot by increments, using the difference quotients

(5) The reason, why we determine the Jacobian J by experimen-tally estimating the partial derivatives of the sensor values to the motion parameters, is the robustness of the control ap-proach in the local working space, i.e. if

is in a suitable bound. Doing so we avoid the very high effort which would be necessary if we would analytically determine the partial derivatives using a large non-linear model. Furthermore the estimates determined by using the real system are expected to be better than the analytical ones, because they are indepen-dent of (inevitable) model abstractions. To get the mapping from the actual sensory pattern to the desired robot’s reference pose no sensor calibration has to be done, because the train-ing process delivers the inverse relation between the robot’s movements and the corresponding sensor values. In the case of programming by virtual demonstration within the graphical simulation environment sufficiently correct world models with respect to sensor and object descriptions must be available for the generation of the controller matrix C. The method is plain and robust.

The simplicity of the programming by demonstration para-digm is fulfilled by an easy way to configure the sensor or shared controlled EM description. The reference sensor pat-tern, i.e. the goal state definition of the atomic EM is directly shown by the current sensor value configuration. The con-troller for achieving this desired state is determined in an automatic manner by the generation of the Jacobian as de-scribed above. The human operator has only to specify the controller input variables, i.e. the sensor combination, which

should be used for the control task, and the controller out-put variables, i.e. the sensor controlled DOFs, to activate the controller generation.

This procedure can be viewed as a form of training by doing. Currently an intelligent sensor planning system is in progress to find out the concerning sensor usage in combination with the determination of the sensor controlled subspaces in an autonomous way [24].

5.4 Sensor fusion aspects

The proposed sensorimotion approach allows a simple inte-gration of different sensor values. An efficient sensor fusion algorithm is inherent implemented by assigning all the used sensor values to the elements of the measurement vector y. To get better estimation results, it is often helpful to use re-dundant and normalized sensor information. We have imple-mented this approach among other cases to combine stereo vision (2 cameras) data with laser range data (4 distance val-ues) and have observed fast (exponentially) as well as wide (full sensor range) convergence.

In the following the main advantages of the proposed con-troller generation process are shortly discussed. The concon-troller matrix C remains constant during the control process. The training of sensor controlled EMs, i.e. to set the linear con-troller coefficients in the Jacobian, has to be done only once for a specific sensor/DOF combination in the desired reference state. The number of training steps for the determination of the controller is equal to the number of DOFs to be controlled. Any constraint space configuration can be selected to integrate the shared control concept into the control generation process. Sensor failures can easily be handled by introducing a binary valued diagonal weight matrix W extending the controller design to "!$#% %& ')( +*, -/. ' 1032 ( 4, (6)

The controller redesign only needs a failure detection, but does not need to apply a new training process for the changed sensor configuration. If is regular then may be regular too, if the original sensor configuration contains enough redundancy. The condition number of

may be used to qualify different sensor configurations.

To realize such a sensorimotion coordination and sensor fu-sion concept we already have applied other methods such as neural networks [25]. Our ongoing work is to compare both approaches with regard to applicability, robustness, and effi-ciency.

6

Experiments

The TSP concept has been verified within ROTEX — an advanced space RObot Technology EXperiment in May ’93 at the D2 spacelab mission [12]. The experimental environment of the ROTEX ground station is used and currently extended to implement our task directed robot programming approach. A powerful man machine interface is one of the key issues to perform the programming by demonstration in an efficient way. A 6DOF control-ball is used to move the robot’s tool center point in the 3D-stereographics simulation. An user-friendly graphical interface based on X and OSF/Motif has been implemented to support the operator in the definition and composition of the various elemental moves. Three types of sensors are available in the real as well as the simulated gripper environment: an array of 8 laser distance sensors with a working range from 0–3 cm, one with 0–30 cm, two 6–axis force-torque wrist sensors (a stiff strain gauge based and a more compliant optical one), and a tiny pair of cameras to provide a stereo image out of the gripper.

The image processing system used in our laboratory is able to extract features like contours or markers in real time (ap-plication dependent). In addition to this direct object feature extraction more abstract measurements such as the Fourier coefficients of a contour can be delivered as an efficient tool for the determination of the pose, size, and shape of an object given by its contour line [26]. The corresponding simulation works analogously on a feature based level with respect to the internal and external camera parameters. Especially for the acquisition of stereo vision data this model based method offers the advantage to avoid the correspondence problem. Corresponding object features in the different camera views can easily be detected by using the same geometric world model interface.

The experimental environment consists of a powerful Silicon Graphics workstation with a multi-processor architecture to enhance the simulation and graphical visualization of the workcell environment. The graphical user interface as well as the controller task are running as independent processes on the same machine. The inter-process-communication occurs via shared memories protected by semaphores to guarantee the mutual exclusion on apparent access collisions.

The exchange of an orbital replaceable unit (ORU) seems to be a good example to explain the task directed programming approach in more detail (see figure 2). This manipulation task can be regarded as a sophisticated pick and place op-eration where the pick includes a bajonet closure screwing action. The composed EM ExchangeOru is split into three more detailed EMs FreeMotion, ApproachToOru, AttachOru, and the inverse EMs FreeMotionWithOru, ApproachWithOru,

ReleaseOru. Let us focus on the EM ApproachToOru. This

composed EM is further divided into three atomic EMs which

differ in the choice of the used sensors and the constraint space configuration.

First, only the vision sensor is available within its measure-ment range. ApproachByVision uses four corner points lying on the ORU contour as the reference pattern to achieve the first goal state. After the execution of the EM ApproachByVision the distance sensors can be applied together with the visible corner points (see figure 3 and 4). The reference state of this EM ApproachByVisionAndDistance is reached if the distance sensor values are zero or non-zero force-torque sensor values appear. The last atomic EM ApproachByDistanceAndForce ends in a state where the attaching of the ORU is possible and the next EM AttachOru can be activated.

FreeMotion AttachOru ApproachToOru ReleaseOru ApproachWithOru FreeMotionWithOru ApproachByVision ApproachByDistanceAndForce Approach ByVisionAndDistance ExchangeOru ApproachToOru

Figure 2 The composed EM ExchangeOru

All three atomic EMs are completely sensor controlled, but for teleoperation the constraint space configuration can easily be reconfigured to allow an operator supported shared con-trol. The Jacobian (see figure 5) for the EM

ApproachByVi-sionAndDistance e.g. integrates the vision data of two visible

corner points of the ORU contour and four distance sensor values. These sensor readings are represented by the sen-sor beams approximately perpendicular to the desired contact surface between the ORU and the gripper in the attach phase (see figure 3 and 4).

z x y tool frame orientation distance sensors stereo camera ORU

y x vision system y x z vision corner points tool frame orientation Figure 4 ApproachByVisionAndDistance, viewed out of the hand camera

We experimentally estimate the Jacobian J by moving the sim-ulated or real robot by increments of 1 mm in each transla-tional DOF and of 2 degrees in each rotatransla-tional DOF, creating the difference quotients

!" # $ (7)

DOF trans(x) trans(y) trans(z) rotate(x) rotate(y) rotate(z) LeftCamX1 -0.23 -4.78 1.32 69.31 -41.98 -139.58 LeftCamY1 5.22 -0.03 0.19 3.47 46.37 -85.95 RightCamX1 -0.14 -4.97 0.82 62.96 -25.91 -146.73 RightCamY1 5.33 -0.03 0.18 2.67 47.73 -85.75 LeftCamX2 0.14 -4.97 -0.82 62.97 25.91 -146.73 LeftCamY2 5.33 -0.03 0.18 -2.67 47.73 85.75 RightCamX2 0.23 -4.78 -1.32 69.31 41.98 -139.58 RightCamY2 5.22 0.03 0.19 -3.47 46.37 85.95 Distance1 0.00 0.00 -1.00 -15.06 10.00 0.00 Distance2 0.00 0.00 -1.00 -15.06 -10.00 0.00 Distance3 0.00 0.00 -1.00 15.06 -10.00 0.00 Distance4 0.00 0.00 -1.00 15.06 10.00 0.00

Figure 5 The Jacobian for the EM ApproachByVisionAndDistance

The incremental values of 1 mm and 2 degrees depend on the geometrical dimensions of workcell and sensor description. We have to consider that the incremental values xk have

to be large enough, to avoid numerical error problems. The linear controller, automatically created and used by the EM

ApproachByVisionAndDistance, generates a smooth behaviour

of the tool center point movements to achieve the desired position.

In the case of programming by demonstration and execution both on the real system the experimental determination of the controller coefficients considers implicitly the sensor charac-teristics. In an analytical approach the particular sensor pa-rameters must be known to get the Jacobian papa-rameters. In the case of programming on the simulated environment and execution on the real the alignment of real and virtual environ-ment have to be guaranteed. Due to the fact that the goal state of the sensor controlled EMs is expressed in terms of sensor

values, the real sensor data must match the sensor values, which are simulated in the virtual environment. For instance the distance sensors have to deliver the correct distance value between the sensor origin and the measured object surface, or the vision system should be able to provide features such as points, lines, or contours in the image plane without lens distortion or something else.

7

Conclusion

We have introduced the TeleSensorProgramming concept as an easy way to program a robot off-line in a robust way with sensor based control structures. The proposed task directed programming paradigm is also applicable in the field of tele-robotics with large time delays where local autonomy is in-dispensable. Teaching by showing in the real as well as in the simulated world, including all the interactions between the robot and its environment, leads to a programming by demonstration approach using an elemental moves concept with local sensor based machine intelligence. The elemental moves are defined by a sensor data processing scheme which is experimentally determined via the Jacobian in the working area respectively.

Our ongoing work is to integrate the virtual and real world by world model update facilities. As the first step we have to enhance the congruence between the virtual and real world using calibrated sensor models.

References

[1] G. Hirzinger, J. Heindl, K. Landzettel, and B. Brunner, “Multisensory shared autonomy — a key issue in the space robot technology experiment ROTEX”, Proc. of

the IEEE Conf. on Intelligent Robots and Systems (IROS), Raleigh, July 7–10, 1992.

[2] T. Sheridan, “Human supervisory control of robot sys-tems”, Proc. of the IEEE Conf. of Robotics and

Automa-tion, San Francisco, 1986.

[3] A. Beiczy and W. Kim, “Predictive displays and shared compliance control for time-delayed telemanipulation”,

IEEE Int. Workshop on Intelligent Robots and Systems, Ibaraki, 1990.

[4] P. Backes and K. Tso, “UMI: An Interactive Supervisory and Shared Control System for Telerobotics”, Proc. of the

IEEE Int. Conf. on Robotics and Automation, Cincinatti, p. 1096–1101, 1990.

[5] J. Funda and R. Paul, “Remote Control of a Robotic System by Teleprogramming”, Proc. of the IEEE Int.

[6] B. McCarragher, “Task-Level Adaptation Using a Dis-crete Event Controller for Robotic Assembly”, IEEE/RSJ

Int. Conf. on Intelligent Robots and Systems (IROS), Yoko-hama, July 26–30, 1993.

[7] A. Kapitanovsky and O. Maimon, “Conceptual Graph-based Synthesis of Robotic Assembly Operations”, Proc.

of the IEEE Conf. on Robotics and Automation, Nice, France, May, 1992.

[8] G. Schrott, “An Experimental Environment for Task-Level Programming”, Second Int. Symposium on

Exper-imental Robotics (ISER), Toulouse, France, June 25–27,

1991.

[9] P. Van de Poel, W. Witvrouw, H. Bruyninckx, and J. de Schutter, “An Environment for Developing and Optimising Compliant Robot Motion Tasks”, Int. Conf.

on Advanced Robotics (ICAR), Tokyo, November, 1993.

[10] E. Ralli and G. Hirzinger, “Fast Path Planning for Robot Manipulators using Numerical Potential Fields in the Configuration Space”, accepted for presentation on the

IEEE/RSJ Int. Conf. on Intelligent Robots and Systems (IROS), Munich, Sept. 12–16, 1994.

[11] M. Mason, “Compliance and force control for computer controlled manipulators”, IEEE Trans. on Systems, Man

and Cybernetics, Vol SMC-11, No. 6, p. 418–432, 1981.

[12] G. Hirzinger, B. Brunner, J. Dietrich, and J. Heindl, “ROTEX — The First Space Robot Technology Experi-ment”, Sixth Int. Symposium on Robotics Research,

Hid-den Valley, PA, Oct 2–5, 1993.

[13] W. Yared and T. Sheridan, “Plan Recognition and Gen-eralization in Command Languages with Application to Telerobotics”, IEEE Trans. on Systems, Man and

Cyber-netics, Vol. 21, No. 2, p. 327–338, 1991.

[14] B. Frommherz and G. Werling, “Generating Robot Action Plans by means of an Heuristic Search”, Proc.

of the IEEE Int. Conf. on Robotics and Automation, Cincinatti, p. 884–889, 1990.

[15] Y. Huang and C. Lee, “An Automatic Assembly Plan-ning System”, Proc. of the IEEE Int. Conf. on Robotics

and Automation, Cincinatti, p. 1594–1599, 1990.

[16] M. Gini, “Integrating Planning and Execution for sensor-based Robots”, in: Ravani B. (ed.), CAD sensor-based

Pro-gramming for Sensory Robots, NATO ASI Series F50, Springer, Berlin, 1988.

[17] S. Tso and K. Liu, “Visual Programming for Capturing of Human Manipulation Skill”, IEEE/RSJ Int. Conf. on

Intelligent Robots and Systems (IROS), Yokohama, July 26–30, 1993.

[18] A. Ude, “Trajectory generation from noisy positions of object features for teaching robot paths”, Robotics and

Autonomous Systems 11, 113–127, 1993.

[19] N. Delson and H. West, “Robot Programming by Human Desmonstration: Subtask Compliance Controller Identi-fication”, IEEE/RSJ Int. Conf. on Intelligent Robots and

Systems (IROS), Yokohama, July 26–30, 1993.

[20] G. Hirzinger and J. Heindl, “Sensor programming — a new way for teaching a robot paths and force/torques simultaneously”, Third Int. Conf. on Robot Vision and

Sensory Controls, 1983.

[21] H. Asada and S. Izumi, “Direct teaching and Automatic Program Generation for the Hybrid Control of Robot Manipulators”, Proc. of the IEEE Int. Conf. on Robotics

and Automation, 1987.

[22] K. Kosuge, T. Fukuda, and H. Asada, “Acquisition of Human Skills for Robotic Systems”, Proc. of the IEEE

Symposium on Intelligent Control, Arlington Virginia, August, 1991.

[23] B. Brunner, G. Hirzinger, K. Landzettel, and J. Heindl, “Multisensory Shared Autonomy and Tele-Sensor-Programming — Key Issues in the Space Robot Tech-nology Experiments ROTEX”, IEEE/RSJ Int. Conf. on

Intelligent Robots and Systems (IROS), Yokohama, July 26–30, 1993.

[24] G. Grunwald and G. Hager, “Towards Task-Directed Planning of Cooperating Sensors”, Proc. of SPIE conf.

on Sensor Fusion V, Boston Mass., Nov., 1992.

[25] G. Wei, G. Hirzinger, and B. Brunner, “Sensorimotion Coordination and Sensor Fusion by Neural Networks”,

IEEE Int. Conf. on Neural Networks, San Francisco,

1993.

[26] K. Arbter and H. Burkhardt, “A Fourier-method for feature extraction and parameter estimation for planar curves in the 3–D space (in German)”, in: