Enhanced Flexibility and Reusability Through State Machine Based Architectures for Multisensor Intelligent Robotics

(1)

Article

Enhanced Flexibility and Reusability Through State

Machine Based Architectures for Multisensor

Intelligent Robotics

Héctor Herrero1,*, Jose Luis Outón1, Mildred Puerto1, Damien Sallé1and Karmele López de Ipiña2

1 _{Tecnalia Research and Innovation, Industry and Transport Division, San Sebastián, Spain;}

[email protected] (J.L.O.); [email protected] (M.P.); [email protected] (D.S.)

2 _{Department of Systems Engineering and Automation, Universidad del País Vasco/Euskal Herriko}

Unibertsitatea, EleKin Research Group, San Sebastián, Spain; [email protected] * Author to whom correspondence should be addressed: [email protected]

Abstract: This paper presents a state machine based architecture which enhances flexibility and reusability of industrial robots, more concretely dual-arm multisensor robots. The proposed architecture, in addition to allowing absolute control of the execution, eases the programming of new applications by increasing the reusability of the developed modules. Through an easy-to-use graphical user interface, operators are able to create, modify, reuse and maintain industrial processes increasing the flexibility of the cell. Moreover, the proposed approach is applied in a real use case in order to demonstrate its capabilities and feasibility in industrial environments. A comparative analysis is presented for evaluating presented approach versus traditional robot programming techniques.

Keywords: intelligent robotics; flexibility; reusability; multisensor; state machine; software architecture; computer vision

1. Introduction

An analysis [1] of the current situation in manufacturing plants highlights 3 major trends:

• An ever increasing customisation of products and short lifecycle, which requires an increase in the flexibility of production means (one unique system must handle all the product diversity and operations) [2,3]. Robots fit perfect in this topic due to their polyvalence; robot programs can adapt to the customisations of the products.

• A large variation in production volumes, which requires an increase in the reconfigurability of production (one system for one product/task within recombinable production lines) [2,4]. Robotic mobile platforms play an important role in this trend, easy to move robots are necessary in some production chains where production volumes change frequently.

• Limited access to skilled operators due to ageing workforce, changes in education and an ever faster technology development. This requires new solutions to assist operators and provide collaborative work environments [5]. Collaborative robotics are being developed for this topic.

The research addressed in this paper focuses on the first trend: the need for highly flexible and intelligent robotic systems. Despite of a large effort in the research community, large companies as well as small and medium enterprises (SME) still don’t have appropriate software tools and solutions to react rapidly with economic viability for an interesting return of investment for the automation of their processes. The direct consequence is that production operations are mostly performed manually, with high operation costs that endanger those companies with respect to lower-wages countries. This research is thus oriented at developing and providing a software ecosystem that allows for a rapid and efficient programming of production processes, providing the required flexibility and permitting an

Preprints (www.preprints.org) | NOT PEER-REVIEWED | Posted: 20 April 2017 doi:10.20944/preprints201704.0130.v1

(2)

effective integration of auxiliary sensors and artificial vision systems. Even if this approach is generic and applicable to industrial manipulators, this paper will be focused on dual-arm multisensor robotic operations.

The dual-arm robots provide more dexterity, in addition to the advantage that they can be used in the existing workstations. Due to these arguments the dual-arm robot deployment is growing year by year, not only in large multinationals, but also in SMEs. Sector experts [2,6] affirm investments for robot deployment are amortised in 1-2 years, but this information cannot be extrapolated to all cases. However, applications with short production batches, environments prone to many changes, and processes that need human-robot collaboration or special environment supervision do not comply with this trend. Dual-arm robots are being introduced in such contexts. The growth of dual-arm systems [7] is resulting in a lot of efforts made by robotic researchers to manage them. Programming, coordinating and supervising bi-manual robots is a need that is increasingly being demanded by the community. Even more with the rise of collaborative robots, which have to integrate different sensors for cell supervising and monitoring [8,9]. In this scenario the need for actuation when external signals are received become essential, e.g., a person enters into the workspace of the robot and it must stop its movement and adapt its behaviour.

In this paper we present an approach to alleviate the challenges that can be identified for dual-arm robotic programming. Presented framework eases the deployment of industrial applications and allows managing the execution control increasing the reliability and traceability of the system (Section2). To ease deployment of this kind of applications we present how the framework can integrate skill based programming. For understanding the advantages, an assembly operation of aeronautical part is detailed. Moreover, an evaluation of the architecture is presented (Section3). Finally, we present discussion, conclusions and future work (Section4and5).

2. Materials and Methods

2.1. State machine based execution coordination for dual arm robots

Traditional robot programming is still not very flexible, thus the dual-arm programming suffers the same problems. In the industry, smaller and smaller batches are ordered, and as a consequence costs of reprogramming the robots grow. Even though there are usually different parts, the process is very similar, e.g., assembling parts with different types of screw. In this case the assembly operation is the same, only the screw size, type or position is changing. Those tasks can be modelled, the key is to be able to subdivide a task (screw operation) into smaller operations (robot movement, end-effector actuation, etc.). Then, re-using these tasks can be made parametrizing correctly the corresponding suboperations without needing to reprogram whole task. Grouping the robot basic movements (primitives) according to tasks or skills is an alternative that many authors have followed [10–14].

One of the most relevant issues in dual-arm robotic programming, especially for industrial applications, is the lack of powerful and easy to use graphical user interfaces [15]. An easy to configure graphical user interface (GUI), which allows the previously mentioned skill based programming, will enable operators to program and maintain the industrial processes. This, in addition to the workers feel themselves part of the automation process, will also contribute to reduce the costs of the robotic systems deployment.

(3)

3 of 19

Figure 1.Proposed overall architecture. It shows how is divided into three levels

designed for coordinating complex systems such as robots. SMACH [22] is another implementation of state machines. It can be defined as task-level architecture for rapidly creating complex robot behaviour. In this work SMACH has been selected for implementing the state machine. One of the reasons is because SMACH can be used under ROS (Robot Operating System) [23,24], which is a flexible framework for writing robot software. ROS is a collection of tools, libraries, and conventions [25] that aim to simplify the task of creating complex and robust robot behaviour across a wide variety of robotic platforms [26]. As a complementary element to the execution control, multi-agent systems can be useful for decision making in coordination and synchronization tasks [27,28].

2.1.1. Proposed architecture

As illustrated in Figure1, the proposed state machine interconnects the application development framework (graphical user interface) and the robotic lower level control system. The presented state machine is composed of different states where each state corresponds with one of the basic operations that the robot can execute. Basic operations are considered the functions or commands that by themselves are able to achieve a goal, e.g., Cartesian point to point interpolation. It can be understood as a robot API (Application Programming Interface). Following the program provided by the user the active state triggers to its corresponding state to execute necessary functions.

In this research all the prototypes are being tested and validated in a dual-arm robot, specifically in a Kawada Nextage Open Robot (Figure2). This robot has humanoid aspect, with two arms of 6 degrees of freedom (DOF) attached to a rotatory torso; it is equipped with a 2 DOF head which incorporates stereo vision system. In conclusion, it is a 15 DOF robot managed by a single controller. In order to obtain more precision another stereo vision systems have been added to each wrist.

As it is detailed in the Section3.1, the applications are composed of tasks and these in turn are composed of primitives (or previously mentioned basic operations) which are translated to states. On one hand, the execution engine triggers state changes at the low level. On the other hand, in the case of Kawada Nextage Open robot, the states are connected to the robotic system through an OpenRTM bridge [29]. But it should not be forgotten that ROS allows hardware independence, and changing the bridge properly, another robotic system can be used (for example, Orocos or Fast Research Interface [30] to interface a Kuka LWR with the proposed architecture).

This combination of the application development framework and a low level state machine allows us to considerably improve the flexibility, hardiness, easier programming, hardware independence and environment control, resulting in a more industry oriented solution.

(4)

Figure 2.Nextage Open Robot where all developments are being tested

2.1.2. Core description

One of the first requirements that was identified was introspection, which is a tool able to provide current execution state continuously, allowing us to manage possible errors and improving the recovery from them. In Figure 3 and Figure 4 the proposed architecture is outlined. The proposed architecture consists of two state machines, one per arm, with some common states. These common states are used when a synchronization between the arms is required, i.e., when both arms of the robot have to move at the same time. This combination of two state machines related by some common states combines the advantages of having individual machines for processes which do not need dual-arm cooperation, with the robustness that allows centralized states for dual-arm requiring processes. The use of SMACH/ROS combination provides some tools that are very useful for introspection. SMACH is using ROS messages for publishing, besides other information, the current state, thus any module of the software can be checked easily.

When the application is launched, the system starts from a Ready state and keeps changing to different states that can be seen as available abilities or capacities of the robot. Note that some states have not been included in order to simplify the diagram. These states are Pause/Stop, Error handling and Finish. The proposed work in this paper allows either human or sensor based supervision of the environment and permits cancelling or adapting plans according to sensor values and perception system information. When an error occurs, e.g., in a trajectory execution, the system is able to cancel the current operation in order to handle the error and return to a safe position (if possible) or enter in alarm state that requires operator intervention.

2.1.3. Description of the developed states

(5)

5 of 19

Figure 3.Proposed state machine based architecture. It represents an overview of the architecture

Figure 4.Proposed state machine based architecture in detail. It shows existing states and transitions.

the available states can be updated by incorporating new capabilities or removing others which will not be used.

In order to understand the proposed architecture, Table1summarizes the different states and their utility. Besides at Table2a summary of the signal and transitions is presented. Each state may contain a more or less complex structure according to its purpose. On the one hand, for example, the Vision Operation state only contains the calls to different vision functions. On the other hand, the Articular/Cartesian motion state is highly general, i.e., this state contains all the required code to manage motions both in Cartesian and articular spaces. For a state transitioning different events are handle, these events can be thrown by safety supervision system or by any module.

2.2. Flexible application development

Proposed architecture in this paper not only refers to the state machine based execution manager, but contains everything necessary for deploying different robotic applications. One of the key advantages of the proposed work is that different applications reuse the common structure of the framework.

2.2.1. Software structure of the framework

In order to ease the maintainability and assure software quality, developed framework is organized in different packages. In this way, following ROS philosophy, each package must fulfil minimum quality criteria.

The simplest application is composed by at least the following three packages: execution engine, core functions and application functions. Figure5illustrates these packages (three columns) and the relation between them. As can be seen execution engine creates (instantiates) the state machines. Each state machine has an instance of an application functions (RivetInstallation,

(6)

Table 1.Summary of the main elements of the state machine

State Description

Ready The state machine is ready for receiving new instructions. This state is waiting until execution engine sends a new request.

Cartesian Articular motion

Manages robot movements both in Cartesian space and articular space. If the movement cannot be executed correctly there is anError handlingstate for manage it.

Full body coordinated

motion

Allows controlling both arms coordinately. Two arms must be in this state to start coordinated motion. Sending the values of the 15 joints of the robot is necessary.

Record trajectory

Allows recording trajectories with a trajectory planner or teaching by demonstration. These trajectories are stored in a database for a future use.

Trajectory execution

Executes trajectories, provided by a trajectory planner or previously stored in a database.

End-effector operation

Manage end-effector operations, depending the end effector different operations can be made, e.g., gripper open/close, deburring tool activate/deactivate, screwing operation, etc.

Vision operation

Manages different computer vision operations. This includes picture acquisition, processing and reference frame transformation among others. As the robotic system has multiple vision systems, this state is responsible to manage them depending on the operation that will be executed.

Master/Slave mode

Puts robot in bi-manual coordinated manipulation mode, one arm actuates as master and the other one as slave. Consists in planning a trajectory for master arm and then computing this trajectory with an offset for the slave arm.

Table 2.Summary of the signals and transitions of the state machine

State Signal Transition to

Ready motion_request Cartesian/Articular motion vision_request Vision operation

end_effector_request End effector operation

... ...

end Finish

Cartesian ok Ready

Articular pause Pause

motion stop Stop

error Error handling

Pause resume Cartesian/Articular motion

stop Stop

error Error handling

Stop error Error handling

Error ok Ready

(7)

7 of 19

Figure 5.Software structure of the framework.

AntenaAssembly...). Application functions inherit from core functions all the attributes and methods, which allows using the robot basic operations (Section 2.1.3), enhancing and particularising them for applying into specific industrial applications. In this way, all applications are composed by core functions (basic operations) and application functions which are a combination of previous ones. These function libraries basically configure the requests for the state machine filling required parameters. This organization also allows having specific graphical user interfaces for each project (rivet_installation_gui) and a common one for basic robot guiding or teaching (dashboard).

2.2.2. Execution engine

Execution engine creates two threads, one per arm; these threads will contain instances of the proposed state machine. Execution engine will continue its execution managing the request of operations, i.e., execution engine is responsible for orchestrating the application flow.

At this point is important to think in the change of paradigm for executing robotic applications. As has been explained here, there are three “independent” threads. As the proposed architecture is running under ROS, the state machine threads are actually ROS nodes, and basically acts like threads with their own parametrisation and independent behaviour. Execution engine communicates with these nodes via ROS messages which contain robot commands with the necessary parametrisation; in this way, each node receives commands to execute and starts triggering the state machine to the convenient state. When the operation is finished the state machine returns to Ready state. The heart of the matter remains in how these messages are generated and managed (Section2.2.3).

The consistency of the execution is guaranteed by the deterministic operation of the state machine. Each node will not receive the next operation until necessary synchronization requirements are met, i.e., until execution engine can assure that state machines are in Ready state. In Section3.1a real use case is presented, Figure12explains graphically how the operations are executed maintaining the coordination of both arms.

2.2.3. Application to executable XML

As commented above, applications are stored in a XML files, but with the particularity that each group of the robot (left arm, right arm and torso/head) has its own instructions. This is because each

(8)

Figure 6.Application program fragment.

state machine needs to execute operations both synchronously and asynchronously: in some cases, a process requires both arms of the robot at the same time, e.g., big part that needs two arms for a correct handling; in other cases, some process can require the use of the both arms but not in the same time. XML files contain, in addition to the operations, the necessary flags and synchronization tools to assure this coordination. In this paper, for the presented use case, the simplest instruction for coordination is used: a wait instruction. This allows one arm waiting until the other arm finishes its ongoing operation.

Generating a simple application (as can be seen in Figure6) can be performed writing each XML file by hand, but when the application and complexity grows is difficult to maintain the correct perspective and timeline, leading to errors. To address this, simple graphical interface can be used. Presented GUI at Figure7obtains a list of available functions from core functions and application functions packages (introduced in Section2.2.1). For creating new applications, the user has to add functions and parametrize them. With the help of the graphical interface many programming errors are avoided, especially for synchronization of both arms, allowing a global vision of the execution flow. In Figure7, at the right frame, application program is represented; the displayed example is for rivet installation process. When application is ready, a XML file is created, containing the list of commands that each arm has to execute. Thewaitfunction represents the simplest synchronization mechanism, because in those time lapse the left arm has to wait until right arm finishes; so in the generated XML this will be translated as wait synchronization operation.

3. Results

3.1. Validation in real use case

(9)

9 of 19

Figure 7.Simple GUI for new application development.

transfer for industry validation of this architecture is currently in process in ReCaM1 [33] project (EU’s Horizon 2020 programme). The relation between a technological centre (Tecnalia), a robotic system integrator (DGH [34]) and the end user (CESA [35]) is a key issue in ReCaM, where the aim to demonstrate a set of integrated tools for the rapid reconfiguration of flexible production systems, particularly the assembly of aeronautical actuators is addressed.

As had been mentioned in previous work [36], one of the most relevant tasks in the aerostructure assembly is the rivet installation operation. In this paper, the progress made on the automation of the riveting installation is presented; in the current prototype a deburring operation has been added, because this prepares the surface of the drilling perimeter for the correct rivet installation. This operation is performed with an integrated deburring tool in one of the grippers of the robot. The other gripper is prepared for taking and introducing rivets into drilled holes. This demonstration takes advantage of the dual-arm capabilities. Furthermore, for robot perception, a stereo vision system has been incorporated for precise hole detection; using incorporated stereo cameras on the arms, production tolerances (0.2mm) can be achieved [37]. In the same way that vision system is used, different kind of sensors can be integrated adding corresponding state to the state machine.

Summarizing, the current demonstrator is composed of the following steps:

a. Detect and debur drilled hole with left arm (Figure8).

b. Pick and extract rivet from a tray with the right arm (Figure9a). c. Insert rivet into detected hole with the right arm (Figure9b).

Note that stepaandbcan be performed at the same time, because rivet extraction operation take more time than deburring operation.

If these operations are viewed as skills, deburring skill, pick rivet skill and rivet inserting skill are obtained. Figure10shows how skills are decomposed in primitives. The organization into skills eases the composition of new programs, because the parametrization is perceptibly easier. This parametrization contains the key features that vary between different skill execution. The way to determine the parameters is as follows: the system programmer start by selecting the references or elements that change for different scenarios. For example, in the case of deburring and insertion, the references of the holes and rivets to be inserted must be parametrized. If that would not be enough,

1 _{This research has received funding from the European Union’s Horizon 2020 research and innovation programme under}

grant agreement no.680759

(10)

Figure 8.A drilled hole is deburred after detecting its position by vision

(11)

11 of 19

Figure 10.Install rivet process organized in skills. Skills are composed by primitives.

the parameters that allow to configure the differences between scenarios would be added. Once these skills have been validated, abstracting from primitives is possible. In the case of deburring operation only the theoretical position must be provided, taking into account that this information can be extracted from the CAD model of the piece.

Calibration or referencing process of the cell is beyond the scope of this work (even if it will be addressed in future work), but can be easily summarized in three steps: at first positions of the drilled holes are obtained, referenced to the origin of the CAD model. After that, using an accurate tool centre point (TCP) three known points of the real piece are touched, the easiest way is usually touching one corner and their adjacent edges with the TCP. With these points position and orientation of the piece can be estimated. Finally using obtained theoretical position of the piece in the robot frame and position of the hole in piece frame a frame transformation can be done to obtain an approximate position of the piece drillings. Of course, this approximate position must be corrected using artificial vision to achieve the required 0.2mm of precision.

Returning to the proposed architecture, once the skills are decomposed, the resulting primitives are the ones that are executed by the state machine. Each state is processing the primitive callbacks, and handling errors if they take place. Thus, the error handling is simpler and it is managed specifically in each state or module. Taking one of the operations that are being analysed, the sequence of the machine state is shown in Figure11.

Execution engine sends to the state machines the request for the next operation, based on the information that is stored in the application XML (see Figure6). The state machine changes from one state to another completing the requested operations. As can be seen in Figure12, some operations of the task of installing one rivet can be performed using both arms of the robot at the same time, improving cycle time. After these coordinated operations, an exclusive movement of the right arm is performed; in this moment left arm is waiting until the right arm finishes the installation of the rivet. The whole process of rivet installation is composed by the repetition of this block of skills. In order to demonstrate the adaptability of the presented framework, CESA [35] use case is presented.

(12)

Figure 11.Debur drilling skill mapping into state machine.

(13)

13 of 19

Figure 13.Assembly skill configuration for one cap of the manifold.

As has been commented above, Tecnalia is working in different projects with assembling operations in aeronautical sector. This use case is being developed under ReCaM project [33], which one of principal topics is the development of assembly capabilities for robots. In ReCaM project the starting point will be the Product Requirement Description, which is first matched against the resource capabilities existing on the current system layout. If no matches are found, the system needs to be reconfigured. New resources can be searched from the resource catalogues. This matchmaking and search is allowed by OWL-based capability model [38], which is used to describe the resource capabilities in a formal, computer- and human-interpretable, manner. The capability matchmaking approach is presented in [39]. Once the system has been re-configured (or found suitable as such), the actual operations need to be programmed and executed. For this programming, the skill-based approach by Tecnalia is utilized. Basically, the required steps are the following: pick and assembly various elements (valves, springs, caps, etc.) into a manifold. All the elements are stored in a kit which can be referenced and located by artificial vision.

In this demonstrator the information extracted from CAD models (offline process that is not in the scope of this work) plays an important role. This information is modelled into different XML files: fixture information and element information. Fixture information XML file contains the position and orientation of the relevant points of the fixture, these points are marked as targets for pick and place operations. Element information XML file contains grasp position, necessary gripper for grasping, and assembly point in the model, i.e., the point that is necessary to align with the fixture relevant point. This skill is able to perform the steps listed above to complete the assembly of different elements into the manifold, only taking into account the information provided in the XML files. Figure13shows a detailed example of how the assembly skill is parametrized using provided information.

As can be seen different application can be modelled following the same schema, the parameters that appears in the skill configuration are codified in a XML file (as has been presented for the previous use case in Figure 6). This XML is completely compatible with the state machine and execution engine (sections2.1.1and2.2.2respectively). The system capacity to adapt to changes in the environment provides advantages. For instance, if there is variation in the position of parts (elements) or in the number of parts to process, the high-level program can be adjusted through minor changes (e.g., reprocess the CAD model for updating positions and adding more blocks of a particular skill). No changes in the low-level program are needed. In consequence, an increase of system flexibility has been achieved.

3.2. Evaluation

In the last years, several methods for evaluating software architectures have been defined: scenario-based (SAAM, ATAM, ALMA, etc.) [40–43], mathematical model-based (Reliability Analysis, Performance Analysis) [44] and metrics-based software architecture evaluation methods

(14)

(QuADAI) [45]. In order to evaluate the advantages of the proposed architecture, based on previously mentioned methods, a simplified approach of Architecture Tradeoff Analysis Method (ATAM) method has been selected to perform a comparative analysis [40,46–48]. When architecture is evaluated, depending of the requirements, different qualities must be analysed. ATAM method concentrates on evaluating suitability, so taking into account appropriate qualities has notable relevancy.

As has been mentioned above, ATAM is a scenario-based method, so different scenarios have been selected in order to compare different desirable qualities. On the one hand, the creation of new application from the beginning scenario has been chosen. New application deployment implies working environment definition, relevant position acquisition, fixture calibration, robot process programming, simulating, testing and adjusting. On the other hand, another common scenario is proposed, adapting an existing application to new product references (the required process would be the same but could change the number of operations or the dimensions of the elements).

The proposed architecture has been compared with different ways of addressing the automation of an industrial process [49,50]. The traditional and most commonly used method is online programming, i.e., teach by demonstration (moving the robot with the teach pendant) replicating the process and acquiring required way points. In other cases, the use of offline programming software can be found. This approach is composed by following steps: generation of 3D scene, tag creation, trajectory planning, process planning, post-processing simulation and calibration [50]. As can be seen the proposed framework in this article is very similar to an offline programming process but with some improvements.

In order to evaluate different approaches, a set of desirable qualities have been analysed: Ease of use. To deal with the first scenario, differences between online programming and other alternatives are evident. A new application deployment requires stopping the production for fixture calibrations, way point acquisition, process replication, simulations and adjustments. These tasks require a high expertise in robotics and programming. With offline alternatives the process can be offline almost entirely, only calibration and final adjustments require stopping the production. Generally offline programming software are very complex and also requires high trained staff. The cost of these technicians (plus license costs) could be not affordable for SME. Proposed framework provides a set of ease-to-configure primitives and skills which reduces the training costs.

Adaptability. This quality impacts in the second scenario. Modifying an existing process using online programming is very time consuming, new position acquisition moving the robot implies stopping the production. For offline programming changes can be made without stopping the production, but depending of the nature of the changes could imply repeating many tasks in order to adapt the application. In the case of proposed approach, the process is similar to offline programming, but with the particularity that the developed skills are programed taking in mind possible changes. For example, in the case of deburring and riveting holes (Section3.1) possible changes in the hole positions and rivet size are anticipated, so the skill takes the information of the hole position and size from a processed CAD file. Then the skill adapts its behaviour configuring target position and gripper aperture to obtained information. The same idea is applied in assembly operation, developed skills can adapt to usual changes in this kind of process: changes in assembly points positions, changes in parts size, etc.

Reliability. Presented approach provides an implicit supervision tools: the state machine allows knowing the current status of the execution. Besides the modular error handling permits an individualized response for the different types of errors. Traditional robot programming techniques require ad-hoc error handling in each critical part of the program.

(15)

15 of 19

Table 3.Strengths and weakness of different robot programming approaches

Quality Online

Programming

Offline Programming

State Machine and Skill Based Programming

Framework

Ease of use - + ++

Adaptability - + ++

Reliability - +- +

Subsetability - + ++

Performance ++ ++

-and abilities is interesting. For the end user, having only the functionalities that are required could reduce the training time and increase the ease of use. Subsetability quality does not exist for traditional online programming, and for offline programming software usually is used only for the commercial aspect.

Performance. Both online programming as offline programming have the best performance, because these methods do not add any layer of software in execution time, i.e., when configuration or set-up phase concluded only a robot specific code is executed in the controller. In the proposed framework XML program is parsed for executing existing skills, which are composed by primitives that execute directly in the robot controller. This, combined with the overhead from the state machine, results in greater demands on processing resources. Even so, executed process and robot movements are the same for all alternatives, so these differences in performance do not affect the overall operation.

Table 2 summarizes the strengths and weakness of different robot programming approaches. Online programming is the simplest approach, which only has the performance as clear advantage. Proposed approach can be seen as enhanced offline programming method, both have in common lot of insights, but through the skill programming and state machine based architecture the ease of use, adaptability and subsetability have been improved. These improvements have the performance drawback, but taking into account the advantages, the tradeoff is acceptable.

4. Discussion

As has been analysed in the previous section, the presented approach in this article offers greater flexibility and reusability (adaptability) than traditional frameworks. On the one hand, the flexibility on this approach is demonstrated by the fact that the same skills can be used to perform different processes although they suffer certain variations, e.g., variations in the rivet models, variations in the drilled holes number or positions, etc. This assertion is supported by the work that the authors have made in different applications [51–54]: another deburring process was performed using very similar skills, antenna assembling skill was presented, workspace monitoring and vision operations for hole detection and 3D CAD Matching were integrated as skills, and finally the interaction between the skills and the state machine was presented. On the other hand, new applications can be generated graphically (Section2.2.3), reducing the requiring expertise and increasing the ease of use. When the user adds a skill to the execution flow, all required parameters must be filled. In this way, a succession of blocks which composes the application is generated. Developed GUI allows exporting sections or entire applications into a XML files in order to increase the re-usability.

One of the foreseen advantages of the present approach is that the state machine architecture can be enhanced with different modules (states) that could be useful in complete different processes. In the proposed scenario the states are related with robot primitives, i.e., robot movements controlled in velocity on the Cartesian space. But the proposed primitives can be combined with nonlinear

(16)

controllers, such as predictive control [55], neural networks or fuzzy approaches [56,57], needed in other industrial processes with high uncertainty in the model like chemical processes (i.e. petrochemical plants). The skills approach could provide additional information and actuation; basic functionality could operate the aperture or closure of valves, and complex implementation could cover other acting elements. It is an idea explored in TOP-REF project [58].

Regarding reliability and robustness that state machine provides, it permits users abstracting from the specifics of dual-arm robotic programming. Proposed framework eases the coordination of both arms with the help of a simple GUI (Figure 7). Besides, a complete traceability of the program status combined with a modular error handling, increase the overall reliability compared with traditional online and offline software.

One of the drawbacks of the presented approach is the performance. The entire ROS ecosystem added to the state machine requires a powerful computer, but taking into account the cost of a computer in relation with an automation project costs, is not a relevant issue. Another relevant topic is that the proposed architecture is hardware agnostic, developed skills are not using robot specific functions, but when primitives are executed, ROS interfaces are used. ROS is compatible with a large number of robots [26], but for an industrial environment ROS-Industrial [59] is more adequate. ROS-Industrial appears with the support of a large research community and robot manufacturers. Their goal is to provide reliable and robust ROS packages. The list of supported industrial robots [60] is growing day by day. This can be a disadvantage compared with available offline programming software, e.g., Delmia, which offers a huge database of robots.

In the industrial world, presenting a framework mostly composed of open source modules always causes discussion. But as has been mentioned in Section 1, nowadays more flexibility and novel solutions are demanded, and open source initiatives as ROS are responding to these requirements of the industry.

5. Conclusions and future work

To improve the control and coordination of anthropomorphic multisensor robots, state machine based architectures have been introduced. This approach allows us to increase the robustness and reliability of the whole system. The proposed architecture is designed to act as a basis for easier programming methodologies. Thanks to the presented graphical user interface, new applications can be generated without the need to be an expert in robotics. With the proper training the operator will be able to create, adapt and maintain industrial processes.

Besides these advantages, the reusability has been noticeably increased. By employing the software architecture that has been presented, completely different applications can leverage well tested modules and functions used in previous developments. At present, the same architecture is being used in different pilot stations with different type of robots and requirements, in these pilot station this technology is under intense tests for validating the usability, robustness and feasibility.

Proposed architecture has been compared with traditional approaches in order to analyse and highlight the strengths and weakness. The next step to follow in the future will be performing a test bench for evaluating and comparing the performance of the robot operation with other alternatives, i.e., online and offline programming. Additionally some stress tests will be applied for assuring the stability of the system.

(17)

17 of 19

Regarding the state machine based architecture, if the proposed approach is used, the industrial processes which can benefit from dual-arm robots are more controlled, and allows an easier and faster deployment of new applications. In the future, the focus will be set on the coordinated manipulation of the arms with the intention of easing this kind of tasks. Besides, the integration of multi-agent system for decision making in coordination and synchronization tasks is being considered.

Acknowledgments:

This research has received funding from the European Union’s Horizon 2020 research and innovation programme under grant agreement no 680759 (ReCaM: Rapid Reconfiguration of Flexible Production Systems).

References

1. Dalal, N.; Triggs, B. Histograms of oriented gradients for human detection. Computer Vision and Pattern Recognition, 2005. CVPR 2005. IEEE Computer Society Conference on. IEEE, 2005, Vol. 1, pp. 886–893. 2. Duguay, C.R.; Landry, S.; Pasin, F. From mass production to flexible/agile production. International

Journal of Operations & Production Management1997,17, 1183–1195.

3. Hu, S.J. Evolving paradigms of manufacturing: From mass production to mass customization and personalization.Procedia CIRP2013,7, 3–8.

4. Wang, W.; Koren, Y. Scalability planning for reconfigurable manufacturing systems. Journal of Manufacturing Systems2012,31, 83–91.

5. Tao, F.; Cheng, Y.; Zhang, L.; Nee, A. Advanced manufacturing systems: socialization characteristics and trends. Journal of Intelligent Manufacturing2015, pp. 1–16.

6. Haslarn, C. The end of mass production? Economy and Society1987,16.

7. Smith, C.; Karayiannidis, Y.; Nalpantidis, L.; Gratal, X.; Qi, P.; Dimarogonas, D.V.; Kragic, D. Dual arm manipulation—A survey. Robotics and Autonomous systems2012,60, 1340–1353.

8. Xia, L.; Chen, C.C.; Aggarwal, J.K. Human detection using depth information by kinect. Computer Vision and Pattern Recognition Workshops (CVPRW), 2011 IEEE Computer Society Conference on. IEEE, 2011, pp. 15–22.

9. Blumrosen, G.; Miron, Y.; Intrator, N.; Plotnik, M. A Real-Time Kinect Signature-Based Patient Home Monitoring System.Sensors2016,16, 1965.

10. Sen, S.; Sherrick, G.; Ruiken, D.; Grupen, R.A. Hierarchical Skills and Skill-based Representation. Lifelong learning, 2011.

11. Thomas, U.; Hirzinger, G.; Rumpe, B.; Schulze, C.; Wortmann, A. A new skill based robot programming language using UML/P Statecharts. Robotics and Automation (ICRA), 2013 IEEE International Conference on. IEEE, 2013, pp. 461–466.

12. Zhou, J.; Ding, X.; Qing, Y.Y. Automatic planning and coordinated control for redundant dual-arm space robot system. Industrial Robot: An International Journal2011,38, 27–37.

13. Andersen, R.H.; Solund, T.; Hallam, J. Definition and Initial Case-Based Evaluation of Hardware-Independent Robot Skills for Industrial Robotic Co-Workers. ISR/Robotik 2014; 41st International Symposium on Robotics; Proceedings of, 2014, pp. 1–7.

14. Vanthienen, D.; De Laet, T.; Decré, W.; Smits, R.; Klotzbücher, M.; Buys, K.; Bellens, S.; Gherardi, L.; Bruyninckx, H.; De Schutter, J. iTaSC as a unified framework for task specification, control, and coordination, demonstrated on the PR2. IEEE International Conference on Mechatronics and Robotics, 2011.

15. Poppa, F.; Zimmer, U. RobotUI-A software architecture for modular robotics user interface frameworks. Intelligent Robots and Systems (IROS), 2012 IEEE/RSJ International Conference on. IEEE, 2012, pp. 2571–2576.

16. Björkelund, A.; Bruyninckx, H.; Malec, J.; Nilsson, K.; Nugues, P. Knowledge for Intelligent Industrial Robots. AAAI Spring Symposium: Designing Intelligent Robots, 2012.

17. Huckaby, J.; Vassos, S.; Christensen, H.I. Planning with a task modeling framework in manufacturing robotics. Intelligent Robots and Systems (IROS), 2013 IEEE/RSJ International Conference on. IEEE, 2013, pp. 5787–5794.

(18)

18. Stenmark, M.; Malec, J. A Helping Hand: Industrial Robotics, Knowledge and User-Oriented Services. AI-based Robotics Workshop, 2013 IEEE/RSJ International Conference on Intelligent Robots and Systems, 2013.

19. Alonso, D.; Vicente-Chicote, C.; Pastor, J.A.; Alvarez, B., Stateml : From graphical state machine models to thread-safe ada code; Reliable Software Technologies - Ada-Europe 2008, Springer, 2008; pp. 158–170. 20. Armentia, A.; Gangoiti, U.; Priego, R.; Estévez, E.; Marcos, M. Flexibility support for homecare

applications based on models and multi-agent technology. Sensors2015,15, 31939–31964. 21. Klotzbuecher, M.https://github.com/orocos/rFSM/tree/master/doc. [accessed on June 2016]. 22. Bohren, J.http://wiki.ros.org/smach. [accessed on June 2016].

23. Quigley, M.; Conley, K.; Gerkey, B.P.; Faust, J.; Foote, T.; Leibs, J.; Wheeler, R.; Ng, A.Y. ROS: an open-source Robot Operating System. ICRA Workshop on Open Source Software, 2009.

24. http://www.ros.org/. [accessed on June 2016].

25. http://www.ros.org/core-components/. [accessed on June 2016]. 26. http://wiki.ros.org/Robots. [accessed on January 2017].

27. Badawy, R.; Yassine, A.; Heßler, A.; Hirsch, B.; Albayrak, S. A novel multi-agent system utilizing quantum-inspired evolution for demand side management in the future smart grid. Integrated Computer-Aided Engineering2013,20, 127–141.

28. Pinto, T.; Vale, Z.; Morais, H.; Sousa, T.M.; others. Strategic bidding in electricity markets: an agent-based simulator with game theory for scenario analysis.Integrated Computer-Aided Engineering2013,20, 335–346. 29. http://openrtm.org/. [accessed on June 2016].

30. http://cs.stanford.edu/people/tkr/fri/html/. [accessed on June 2016]. 31. http://www.tecnalia.com/en/, accessed on 2017. [accessed on February 2017]. 32. http://www.project-leanautomation.eu/. [accessed on June 2016].

33. http://recam-project.eu/. [accessed on February 2017].

34. http://www.grupodgh.es/en/, accessed on 2017. [accessed on February 2017]. 35. http://www.cesa.aero/en/, accessed on 2017. [accessed on February 2017].

36. Herrero, H.; Outón, J.L.; Esnaola, U.; Sallé, D.; de Ipiña, K.L. State Machine Based Architecture to Increase Flexibility of Dual-Arm Robot Programming. InBioinspired Computation in Artificial Systems; Springer, 2015; pp. 98–106.

37. Herrero, H.; Esnaola, U.; Sallé, D.https://www.youtube.com/watch?v=pvxlqyJtPNo. [accessed on April 2017].

38. Järvenpää, E.; Siltala, N.; Lanz, M. Formal Resource and Capability Descriptions Supporting Rapid Reconfiguration of Assembly Systems. In Proceedings of the 12th Conference on Automation Science and Engineering, and International Symposium on Assembly and Manufacturing. IEEE, 2016, pp. 120–125. 39. Järvenpää, E.; Siltala, N.; Hylli, O.; Lanz, M. Capability matchmaking procedure to support rapid

configuration and re-configuration of production systems. Submitted for publication in 27th International Conference on Flexible Automation and Intelligent Manufacturing, FAIM2017, 2017.

40. Babar, M.A.; Zhu, L.; Jeffery, R. A framework for classifying and comparing software architecture evaluation methods. Software Engineering Conference, 2004. Proceedings. 2004 Australian. IEEE, 2004, pp. 309–318.

41. Dobrica, L.; Niemela, E. A survey on software architecture analysis methods.IEEE Transactions on software Engineering2002,28, 638–653.

42. Ionita, M.T.; Hammer, D.K.; Obbink, H. Scenario-based software architecture evaluation methods: An overview. Icse/Sara2002.

43. Kazman, R.; Klein, M.; Clements, P. ATAM: Method for architecture evaluation. Technical report, DTIC Document, 2000.

44. Cheung, L.; Roshandel, R.; Medvidovic, N.; Golubchik, L. Early prediction of software component reliability. Proceedings of the 30th international conference on Software engineering. ACM, 2008, pp. 111–120.

45. Gonzalez-Huerta, J.; Insfran, E.; Abrahão, S.; Scanniello, G. Validating a model-driven software architecture evaluation and improvement method: A family of experiments. Information and Software Technology2015,57, 405–429.

(19)

19 of 19

47. Giorgini, P.; Kolp, M.; Mylopoulos, J. Multi-agent and software architectures: a comparative case study. International Workshop on Agent-Oriented Software Engineering. Springer, 2002, pp. 101–112.

48. Ringert, J.O.; Rumpe, B.; Wortmann, A. A Case Study on Model-Based Development of Robotic Systems using MontiArc with Embedded Automata. arXiv preprint arXiv:1408.56922014.

49. Biggs, G.; MacDonald, B. A survey of robot programming systems. Proceedings of the Australasian conference on robotics and automation, 2003, pp. 1–3.

50. Pan, Z.; Polden, J.; Larkin, N.; Van Duin, S.; Norrish, J. Recent progress on programming methods for industrial robots. Robotics and Computer-Integrated Manufacturing2012,28, 87–94.

51. Herrero, H.; Outon, J.L.; Esnaola, U.; Salle, D.; Lopez de Ipina, K. Development and evaluation of a Skill Based Architecture for applied industrial robotics. Bioinspired Intelligence (IWOBI), 2015 4th International Work Conference on. IEEE, 2015, pp. 191–196.

52. Herrero, H.; García, F.; Esnaola, U.; Sallé, D. https://www.youtube.com/watch?v=x-eJ66jM1Rk. [accessed on April 2017].

53. Herrero, H.; Moughlbay, A.A.; Outón, J.L.; Sallé, D.; de Ipiña, K.L. Skill based robot programming: Assembly, vision and Workspace Monitoring skill interaction. Neurocomputing2017. Accepted for publication. Available online at: http://doi.org/10.1016/j.neucom.2016.09.133.

54. Herrero, H.; Pacheco, R.; Alberdi, N.; Rumayor, M.; Salle, D.; Lopez de Ipiña, K. Skills for vision-based applications in robotics application to aeronautics assembly pilot station. EUROCON 2015-International Conference on Computer as a Tool (EUROCON), IEEE. IEEE, 2015, pp. 1–6.

55. Wang, T.; Gao, H.; Qiu, J. A Combined Fault-Tolerant and Predictive Control for Network-Based Industrial Processes. IEEE Transactions on Industrial Electronics2016,63, 2529–2536.

56. Wang, T.; Zhang, Y.; Qiu, J.; Gao, H. Adaptive fuzzy backstepping control for a class of nonlinear systems with sampled and delayed measurements. IEEE Transactions on Fuzzy Systems2015,23, 302–312.

57. Wang, T.; Qiu, J.; Gao, H.; Wang, C. Network-Based Fuzzy Control for Nonlinear Industrial Processes With Predictive Compensation Strategy.IEEE Transactions on Systems, Man, and Cybernetics: Systems2016. 58. http://toprefproject.eu/. [accessed on April 2017].

59. http://rosindustrial.org/about/description/. [accessed on January 2017]. 60. http://wiki.ros.org/Industrial/supported_hardware. [accessed on January 2017].

c

2017 by the authors. LicenseePreprints, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC-BY) license (http://creativecommons.org/licenses/by/4.0/).