This paper addresses an open issue in autonomous soft- ware for robotics platforms: how to perform intensive ex- periments according to some structured methodology. We have first produced the OGATE framework to support au- tomated testbench campaigns to achieve performance mea- sures of different autonomous controllers. This framework is composed of (i) a methodology to define and guide the testing process, and (ii) an engineering tool to support such a methodology. The paper in particular introduces a method-
ology for evaluating deliberative robotic components and a way for compact visualising the result of its application. Us- ing the new framework we have analyzed different planning policies for the deliberative component of the GOACsystem here considered as an external system. Assessment has in- volved inspecting internal measures of the deliberative com- ponent while executing scenarios with increasing complex- ity. Within these tests we have been able to obtain differ- ent reports describing the performance of the system that al- lows us to obtain conclusions that were hard to be achieved performing standalone tests. In particular for the extensive GOACtesting we have been able to identify an issue in the single goalpolicy of the planning component, and we have been able to conclude that the all goals policy is open to dynamic controllability issues.
Some specific issues still require some additional com- ments. First, the need of considering also stochastic model- ing and, more in general, to allow for setting up more com- prehensive and complex testing scenarios is a further de- sired feature. As for assessment of plan-based components, OGATE is a rather new solution and then the present pa- per is more focused on proposing an evaluation methodol- ogy and a technological supporting tool. Indeed, as a start- ing point we are considering a set of preordered scenarios from a real world robotic application (hence emphasizing the realism of the use case). Nevertheless, OGATE can be seamlessly extended in order to consider more complex sce- narios, e.g., including stochastic features and to allow users performing more thorough test campaigns analysis. This is actually an ongoing work.
Then, the choice of relying on plugins enables OGATE to connect to any kind of control architecture notwithstanding which kind of robotic platform/software is deployed. Thus, OGATE aims to be as much as possible platform/software independent. Nevertheless, the implementation of a ROS plugin is in our agenda for future versions of the framework. In order to develop a plugin to connect OGATE to an autonomous controller, some technical work by skilled engineers and/or control architecture experts is obvi- ously required. Nevertheless, after such mandatory step, OGATE is requiring system users (i.e., not necessarily ex- perts/engineers) to generate two XML files for describing i) the set of components necessary to actually execute the au- tonomous controller and ii) the metrics to be measured as well as the data required to generate the evaluation reports. Both files can be generated through the graphical environ- ment of OGATE2. The file contains detailed information
(order of execution, paths, processes, parameters, simulator settings file, etc.) about software processes to be issued in order to properly activate the GOAC and the simulator soft- ware.
Finally, among future developments, a further analysis of robot missions evaluation is required to leverage OGATE tool in order to identify relevant metrics to evaluate control architectures also allowing metrics customization according
2
As an example, at the following link, an example of an XML file with the information to execute the GOAC controller is pro- vided: https://www.dropbox.com/l/UBKryBFRaZq7taCzdUFCso.
to specific robot mission requirements. In general, the defi- nition of a more thorough set of standard metrics is highly desirable and would constitute a not trivial and important contribution in this field. Moreover, the use of OGATE will be considered for comparing different plan-based delibera- tive platforms on the same benchmark tests.
Acknowledgments
Pablo Mu˜noz is supported by the European Space Agency (ESA) under the Networking and Partnering Initiative. UAH authors are partially supported by the Junta de Comunidades de Castilla-La Mancha project PEII-2014-015-A. CNR au- thors are partially supported by the Italian Ministry for Uni- versity and Research (MIUR) and CNR under the GECKO Project (Progetto Bandiera “La Fabbrica del Futuro”). Au- thors want to thank to the ESA’s technical officer Mr. Michel Van Winnendael for his continuous support.
References
Ad Hoc ALFUS Working Group. 2007. Autonomy Levels for Unmanned Systems (ALFUS) Framework – Framework Models. Technical Report 1011-II-1.0, National Institute of Standards and Technology.
Alami, R.; Chatila, R.; Fleury, S.; Ghallab, M.; and Ingrand, F. 1998. An architecture for autonomy. Field Robotics, Special Issue on Integrated Architectures for Robot Control and Programming17:315–337.
Aschwanden, P.; Baskaran, V.; Bernardini, S.; Fry, C.; R- Moreno, M. D.; Muscettola, N.; Plaunt, C.; Rijsman, D.; and Tompkins, P. 2006. Model-unified planning and execution for distributed autonomous system control. In Association for the Advancement of Artificial Intelligence (AAAI) 2006 Fall Symposia.
Behnke, S. 2006. Robot competitions – ideal benchmarks for robotics research. In 2006 IEEE/RSJ International Con- ference on Robots and Systems (IROS) Workshop on Bench- marks in Robotics Research.
Bensalem, S.; de Silva, L.; Gallien, M.; Ingrand, F.; and Yan, R. 2010. “Rock Solid” Software: A Verifiable and Correct- by-Construction Controller for Rover and Spacecraft Func- tional Levels. In i-SAIRAS-10. Proc. of the 10thInt. Symp.
on Artificial Intelligence, Robotics and Automation in Space. Ceballos, A.; Bensalem, S.; Cesta, A.; Silva, L. D.; Fratini, S.; Ingrand, F.; Oc´on, J.; Orlandini, A.; Py, F.; Rajan, K.; Rasconi, R.; and Winnendael, M. V. 2011. A Goal-Oriented Autonomous Controller for Space Exploration. In ASTRA 2011 - 11th Symposium on Advanced Space Technologies in Robotics and Automation.
Cesta, A.; Cortellessa, G.; Fratini, S.; and Oddi, A. 2009. Developing an end-to-end planning application from a time- line representation framework. In IAAI-09. Proc. of the The Twenty-First Innovative Applications of Artificial Intel- ligence Conference.
del Pobil, A. P. 2006. Why do we need benchmarks in robotics research? In 2006 IEEE/RSJ International Confer- ence on Robots and Systems (IROS) Workshop on Bench- marks in Robotics Research.
tonomous robotics. In Amigoni, F., and Schiaffonati, V., eds., Methods and Experimental Techniques in Computer Engineering, SpringerBriefs in Applied Sciences and Tech- nology. Springer International Publishing. 55–68.
Fratini, S.; Pecora, F.; and Cesta, A. 2008. Unifying Plan- ning and Scheduling as Timelines in a Component-Based Perspective. Archives of Control Sciences 18(2):231–271. Gat, E. 1992. Integrating planning and reacting in a het- erogeneous asynchronous architecture for controlling real- world mobile robots. In the Tenth National Conference on Artificial Intelligence (AAAI), 809–815.
Gat, E. 1997. On Three-Layer Architectures. In Artificial Intelligence and Mobile Robots. MIT Press.
Gertman, D. I.; McFarland, C.; Klein, T. A.; Gertman, A. E.; and Bruemmer, D. J. 2007. A methodology for testing un- manned vehicle behavior and autonomy. In Performance Metrics for Intelligent Systems (PerMIS’07) Workshop. Huang, H.-M.; Messina, E.; Jacoff, A.; Wade, R.; and Mc- Nair, M. 2010. Performance measures framework for un- manned systems (PerMFUS): Models for contextual met- rics. In Performance Metrics for Intelligent Systems (Per- MIS’10) Workshop.
Hudson, A. R., and Reeker, L. H. 2007. Standardizing mea- surements of autonomy in the Artificially Intelligent. In Per- formance Metrics for Intelligent Systems (PerMIS’07) Work- shop.
McWilliams, G. T.; Brown, M. A.; Lamm, R. D.; Guerra, C. J.; Avery, P. A.; Kozak, K. C.; and Surampudi, B. 2007. Evaluation of autonomy in recent ground vehicles using the autonomy levels for unmanned systems (ALFUS) frame- work. In Performance Metrics for Intelligent Systems (Per- MIS’07) Workshop.
Morris, P., and Muscettola, N. 2005. Temporal dynamic controllability revisited. In In Procs. of the 20th National Conference on Artificial Intelligence (AAAI-2005).
Mu˜noz, P.; Cesta, A.; Orlandini, A.; and R-Moreno, M. D. 2014. First steps on an on-ground autonomy test envi- ronment. In 5th IEEE International Conference on Space Mission Challenges for Information Technology (SMC-IT). IEEE.
Nesnas, I.; Simmons, R.; Gaines, D.; Kunz, C.; Diaz- Calderon, A.; Estlin, T.; Madison, R.; Guineau, J.; McHenry, M.; Shu, I.-H.; and Apfelbaum, D. 2006. CLARAty: Chal- lenges and steps toward reusable robotic software. Advanced Robotic Systems3(1):23–30.
Oreb¨ack, A., and Christensen, H. I. 2003. Evaluation of architectures for mobile robotics. Journal of Autonomous Robots14:33–49.
Parasuraman, R.; Sheridan, T. B.; and Wickens, C. D. 2000. A model for types and levels of human interaction with au- tomation. Systems, Man and Cybernetics, Part A: Systems and Humans, IEEE Transactions on30(3):286–297. Poulakis, P.; Joudrier, L.; Wailliez, S.; and Kapellos, K. 2008. 3DROV: A planetary rover system design, simula-
SAIRAS).
Py, F.; Rajan, K.; and McGann, C. 2010. A System- atic Agent Framework for Situated Autonomous Systems. In AAMAS-10. Proc. of the 9th Int. Conf. on Autonomous