In this chapter we presented and analyzed different fully integratedmodel-basedsystem architec- tures. We showed that already locally reactive control which integrates perceptual feedback at the highest possible rate could be very efficient for simple known tasks while being safe by avoiding collisions with the environment. Reactive Planning achieves a better performance in more complex environments due to its ability to look ahead. Knowledge about the kinematicmodels
is essential for the presented system to reason about the future system behavior and achieve its safe and reactive behavior. Sense-plan-act performs well in static scenarios as expected, but even there locally reactive control and reactive planning have advantages as they can start moving earlier while continuing to consume feedback.
We also observed that on the trade-off curve between perceptual accuracy and computational speed, it is more beneficial to have fast feedback than accurate world representations. This is
especially the case for dynamic and uncertain manipulation scenarios where a fast reaction to sudden changes or new incoming information is critical. As communication bandwidth is limited, this also places constraints and how much information can be transferred between components. Therefore, we opted formodel-basedvisual tracking and querying SDFs only for a small subset of points on the robot. Data association, identifying if observed points are part of the robot or the environment, was significant and we therefore carefully synchronized the different sensory and information streams across the three different computers. Despite synchronization, heuristics for data association are a current limitation.Learning-basedapproaches are an ideal candidate to replace these heuristics and infer whether or not to ignore voxels in the occupancy grid. The presented system still solves relatively simple pick and place tasks, albeit going beyond state-of-the-art operating in dynamic environments. Constraining ourselves to simpler tasks allows to formulate robust observation models and well-defined cost functions capturing collision avoidance tasks. By choice, the presented system does not rely on any learning to tackle the different scenarios. However, based on the experience with thismodel-basedsystem we can identify several avenues forlearning-basedextensions. The current architecture operates on visual and joint encoder feedback. However, manipulation tasks are heavily concerned with contact interaction. We use the finger strain gauges as feedback in the grasp controller. Our system would further benefit from also taking haptic feedback from tactile sensor arrays or force/torque sensors into account. Modellingthese sensor modalities is challenging. Fusing multiple measurements from very diverse sensors and using them for execution feedback is a challenging problem. To the best of our knowledge, no complex fullymodel-basedrobotic manipulation system exists, capable of handling a diverse set of sensor modalities for contact interactions. In Chapter 4 we therefore present a novellearning-basedapproach to tackle this problem.
To empirically quantify the importance of fast feedback, we had to make strong assumptions about the manipulation task. For instance, only known objects with manually predefined grasps have been considered to allow for a fair comparison between the different methods. Albeit robots often have to interact with a small set of known objects, automatically inferring good grasp poses for unknown objects would be essential for the presented system to achieve a higher level of autonomy.Model-basedmethods for grasping are known to not translate well to the real world due to model inaccuracies and computation requirements (Chapter 5). Therefore, we propose
learning-basedmethods capable of extending the discussed system by replacing the manually defined grasps for known objects with a deep neural network (Chapter 5), capable of inferring grasps poses from partial sensory observation even for unknown objects.
Another key assumption for the success of our reactive manipulation was precise and fast trajectory tracking. Although our system achieves good tracking using model-basedinverse
dynamics and PID control (Chapter 2.1.4), it requires high feedback gains to allow for unmodeled changes to the robot’s dynamics when grasping and lifting objects. Because of the real-time perceptual feedback our presented manipulation system proactively avoids obstacles, thus, is very safe. However, due to high feedback gains required to compensate formodelinaccuracies while grasping, it is not as compliant as possible, therefore not inherently safe. Chapter 6 discusses a learning-basedmethod which is agnostic to the underlyingmodel-basedinverse dynamics method and improves its performance from data. This approach can further improve the fully integrated system, discussed in this chapter, without sacrificing domain specific robotics knowledge forlearning.
Online Decision-Making for Manipulation
Complex and dynamic environments require to close feedback control loops as demonstrated in the previous Chapter 3. Within this thesis, we have already shown that closing perceptual feedback loops allowmodel-basedsystem architectures to perform challenging pick and place manipulation tasks while being agnostic to continuous environmental changes. However, here we argue that robust task execution for complex interaction tasks, beyond pick and place, requires to close feedback control loops in novel ways around many more sensory signals than done traditionally. Without considering different sensor modalities, the robotic system cannot cope with noise and uncertainty in the sensory-motor system of the robot and the environment during contact rich tasks because important events cannot be observed. For instance, when unscrewing the cap of a bottle force/torque measurements allow detecting if the cap becomes lose which is very difficult to infer from perceptual data. Low-level control systems increase the execution robustness by integrating feedback of high-bandwidth sensors, e.g., force/torque. Still, task execution often fails due to external perturbations that are hard to observe andmodelin the low- level control system, using high-bandwidth sensors. Hereafter, we propose to close a high-level feedback loop, leveraging additional low- and high-bandwidth sensor modalities increasing the task robustness usinglearning.
One challenge towardsautonomous manipulationis to provide feedback controllers with appro- priate reference signals and decide onlinewhento consider alternative behaviors to counteract perturbations.Model-basedreactive systems (see Chapter 3) have shown promising results for known manipulation tasks as long as their models are correct and no complex contact interactions are required. When such complex contact interactions are necessary, for instance, complex grasp- ing interactions, modeling errors are inevitable and will significantly degrade the performance of themodel-basedplanner (Weisz et al. 2012), e.g., for dexterous object manipulation. Uncertainty about the state of the manipulated object increases exponentially fast and therefore planning becomes intractable, unless smart heuristics/biases (Brock 2011) are applied.
Figure 4.1:The ARM-S robot manipulating a bottle.
Additionally, model-based methods (Chapter 3) require proper task-specific objectives and programming. Here, we present a novellearning-basedapproach, treating manipulation tasks as a data-driven sequential online decision-making problem, bootstrapped from demonstrations. We do not neglect allmodel-based information, for instance, the perceptual system uses the same model-based probabilistic object tracker as before. To effectively use learning from demonstration, we leverage the fact that most manipulation tasks, e.g., unscrewing a bottle (see Figure 4.1), locally decompose into a sequence of skills. A sequence can be encoded in a state machine, termedmanipulation graph, see Figure 4.3. Conceptually this graph is similar to the task decomposition used in the previous Chapter 3 with the main difference that we merely have to demonstrate skills and associate their high-level interconnection rather than determining objectives formodel-basedoptimization-based planning and locally reactive control. This graph representation imposes constraints onto the possible sequence of manipulation skills, similar to how a grammar constraints the possible sequences of words to form sentences. Such a representation can be either inferred from data (Kroemer et al. 2014; Niekum et al. 2013) or provided by human operators, who usually have good intuition about the necessary high-level information and skill decomposition to accomplish a certain task. However, manually designing rules, based on current sensory information to determine whether or not the current skill execution is valid and where to start an alternative skill, is very hard. This is mainly due to differences in sensor modalities, noise characteristics, and the high dimensionality of the signal space.
decision-making problem into two related classification problems. Both problems have to operate in real-time to close the high-level feedback loop, learning whento switch skills andwhich
skill to execute next. Sensory experiences, stored in themanipulation graphstructure, contain all required information and should allow autonomous data extraction, only requiring user interventions at failure conditions.
Task-relevant perceived sensor signals are strongly correlated with the executed manipulation action (Pastor et al. 2011b) is the critical insight enabling thislearning-basedsystem. Thus, given similar task execution, the robot will perceive similar sensory feedback (Pastor et al. 2011a). Therefore, skills used throughout this chapter are encoded as Associative Skill Memories (ASMs), sensor information encoded in sync with movement primitives, as introduced in (Pastor et al. 2012). Model-basedplanning approaches (Chapter 3) address a more general global solution space, making it challenging to associate sensory traces with their execution. However, these associated signals are essential to training our classification-based online decision-making system (ODMS). To use stereotypical movements, we focus on solving a particular manipulation task instance in this chapter, unscrewing a bottle cap and removing it and screwing it back on. Note, the presented approach could also be used to demonstrated robust grasping behavior, bridging the gap between perceptual grasp pose prediction (Chapter 5) and manual grasp controller design (Chapter 3).
The development and discussion of thelearning-basedODMS is the main contribution of this chapter. It decides in real-time, at any moment of the task execution, if the currently executed skill should be replaced with another one (when) and which skill to choose as a replacement (which). We only require human demonstration to learn stereotypical movements and intervention, stop task execution in case the system is about to fail. These real robot skill demonstrations and execution interventions result in a dataset, stored in themanipulation graphstructure. ODMS automatically extracts supervised datasets from the real-robot experiments allowing for training our classification based decision-making system. Our formulation enables integrating sensors even if they contain no valuable information about the task. The importance of different sensor modalities at different execution stages is automatically inferred. Thus, manual task-specific feature selection is less crucial. We further report qualitative results of our method applied to a dexterous manipulation task performed by a bimanual robotic system, see Figure 4.1.
4.1 Related Work
Motion planning approaches such as (N. Ratliff et al. 2009; Kalakrishnan et al. 2011a) are mostly applied to problems neglecting object interactions, see Chapter 2.1.3, 3.1, and 3.4 for a more thorough discussion. One recent approach that creates plans for contact manipulation, proposed in (Dogar et al. 2010), still uses a quasi-static assumption to obtain a feasible search space. This work builds upon results reported in (Pastor et al. 2011a,b, 2012), a skill-based formulation towards autonomous manipulation. In contrast to (Pastor et al. 2012), we propose to close the high-level feedback loop by running a decision-making process online at all time and not only selecting the next ASM at the terminal condition of the current ASM. Different from (Pastor et al. 2011a), we propose tolearnfailure case prediction from a supervised signal, obtained from the ASMs organized in themanipulation graphstructure. Additionally, we reduce the number of open hyper-parameters, compared to the previous work by (Pastor et al. 2011a). The
manipulation graphis capable of generating a large number of skill sequences. This structure is assumed to be manually provided by a user similar to the manually designed task structure in Chapter 3. A manipulation graph could be constructed automatically, as proposed in (Kroemer et al. 2014; Niekum et al. 2013). However, the construction process is an open research problem by itself. (Kroemer et al. 2014) proposed to infer the hidden phases of the manipulation task from several trial executions, using an autoregressive Hidden Markov model. They train a logistic classifier (see Chapter 2.2.2.3), on a small, manually selected set of discriminative features to model the transition probability deciding when to switch between states in their inferred Hidden Markov model. (Niekum et al. 2013) present another approach to automatically infer a state machine representation which can be incrementally refined. Similar to (Pastor et al. 2012), this approach is limited to skill switching at the end of each skill execution.