Ricercatore: Ilya Afanasyev Soggetto
ospitante: UNIVERSITA' DEGLI STUDI DI TRENTO
Bando: Incoming post doc 2009
Soggetto partner (solo per outgoing):
e-mail: ilya.afanasyev@unitn.it, ilya.afanasyev@gmail.com
Ai sensi della Decreto Leg. n. 196 del 30/06/2003 – Codice in materia del trattamento dei dati personali - autorizzo la pubblicazione sul Sito internet della PAT dell’abstract e delle immagini (foto o filmati) relativi al mio progetto.
Area di ricerca: Mechathronics / Meccatronica Acronimo Laser-Camera systems
Titolo Laser-Camera systems and algorithms to sense the environment for robotic applications Data inizio 12 maggio, 2010 Durata 36 Finanziamento: € 149.000,00
Provincia Autonoma di Trento European Union – 7th research framework programme Marie Curie Actions – COFUND
“TRENTINO”
- The research, training and mobility programme in Trentino -
PCOFUND-GA-2008-226070
The final goal of the project is to build and mount a system of a laser rangefinder (LRF) and a camera (oriented and calibrated together with LRF) on a robotic vehicle, using them together with other 3D sensors and concentrating on Robotic and Human-Machine Interface (HMI)
applications by creating algorithms to sense the environment. The general purposes of the project are:
1. Building a sensor system, including the Laser-Camera (LCS), for robotic applications. 2. Mounting LCS on the robotic vehicle (initially – on a prototype of a robotic vehicle, then - on the robot Pioneer P3-DX).
3. Development of objects recognition and localization algorithms, based on processing 3D point cloud obtained from sensor system (LCS, Kinect, camera, etc.).
4. Verification of the algorithms with hardware facilities, including tests with different 3D sensors and the robot “Pioneer P3-DX”.
The sensor system includes the laser rangefinder (LRF), RGB camera and also an additional sensor (Microsoft Kinect for data validation). This sensor system captures a target object as 3D point cloud, which is structured and processed by object localization and recognition algorithms together with simultaneous processing of images from 2D camera. For sensor fusion the sensor system was mutually calibrated, using the standard camera calibration technique (Bouguet method). Microsoft Kinect sensor was applied to (a) validate LCS measurements, and (b) control accuracy and precision of object location. The joint 3D and 2D data are processed by the state-of-the-art object recognition and localization algorithms, which can be used for path planning /
mapping the indoor robots (based on sensor integration of RGB camera and a 2D laser rangefinder) and also for different scientific and industrial applications. For these purposes:
- The Sensor System has been investigated by indoor environmental conditions of the
Mechatronics lab., UniTN, both independently and together with the robot “Pioneer P3-DX” (after on-board installation).
- The object recognition and localization algorithms have been created.
- The sensor data has been validated and the measurement accuracy and precision have been controlled with Microsoft Kinect.
The main accomplishments obtained by the project are:
- Building the Laser-Camera system for robotic applications (Fig.1).
- Mounting laser-camera system on the robot “Pioneer P3-DX” (see Fig. 2).
- Development and experimental verification of 3D objects recognition and localization algorithms (with different sensors and hardware facilities, Fig.3,4).
- Improvements of the object localization technique, achieving the measurement accuracy and precision comparable with Microsoft Kinect (Fig.5,6).
- The implementation of the algorithms with hrdwre facilities to enhance the motion control with object and environment mapping (Fig.7,8).
Figure 1. A laser and a camera oriented/calibrated together form the Laser-Camera system.
Figure 2. Robotic vehicle with Laser-Camera System (LCS).Left -the prototype of a robotic vehicle with installed LCS (before purchasing the robot in 2012). Right – LCS installed on the robot “Pioneer P3-DX”.
Provincia Autonoma di Trento European Union – 7th research framework programme Marie Curie Actions – COFUND
“TRENTINO”
- The research, training and mobility programme in Trentino -
PCOFUND-GA-2008-226070
Figure 3. Human pose recognition: the block diagram of 3D Human Body Pose algorithm (left), the result pose (right): photo, 3D point cloud, simulation in superquadrics.
Figure 4. 3D gesture recognition. Data acquisition (left) and the result of gesture pose estimation by Superquadrics (right).
Figure 5: Experiment setup layout, data were acquired both from a camera and Kinect. Chessboard is used to estimate camera and Kinect positions to provide the reference frame for the localization.
Figure 6: The comparison between the ellipses of uncertainty (green - for Kinect, blue - for new system) for the box (left) and the cylinder (central) localizations according to the reference grid. SuperQuadrics fitting with depth data of a cylinder and a box (right).
Possible impacts. 3D Laser-Camera System can provide 2D/3D measurements capturing 2D/3D data of an object (with distance and color information) for indoor applications.
Object recognition and pose estimation algorithms can be applied for:
- detection and reconstruction by math. models of different real objects (including complex objects, like human body, gesture recognition, etc.).
-detection, classification and selection a definite thing in a heap of different objects for robotic manufacturing (such as a search an object in mechanical details laying in mess).
Provincia Autonoma di Trento European Union – 7th research framework programme Marie Curie Actions – COFUND
“TRENTINO”
- The research, training and mobility programme in Trentino -
PCOFUND-GA-2008-226070
Figure 7: The set of some training images (for a cylinder and a box) and the final results of detection (the red rectangle surrounding the objects for test images).
Publications.
1. Afanasyev I. and De Cecco M. 3D Gesture Recognition by Superquadrics. // Proc. VISAPP conf. (Barcelona, Spain), V.2, 429-433 (February, 2013). ISBN: 978-989-8565-48-8.
(www.ing.unitn.it/~afanasye/Publications/VISAPP_2013/)
2. Enrico Zappia, Ilya Afanasyev, Nicolo' Biasi, Mattia Tavernini, Alberto Fornaser, Antonio Selmo, and Mariolino De Cecco. Monocular object localization by superquadrics curvature reprojection and matching. // Proc. Graphicon-2012 conf. (Moscow, Russia), 33-38 (October, 2012). (http://www.ing.unitn.it/~afanasye/Publications/Graphicon_2012/)
3. Afanasyev I., Lunardelli M, De Cecco M., et al. 3D Human Body Pose Estimation by
Superquadrics. // Proc. VISAPP conf. (Rome, Italy), V.2, 294-302 (February, 2012). ISBN: 978-989-8565-04-4. (http://visapp.visigrapp.org/Abstracts/2012/VISAPP_2012_Abstracts.htm) 4. Afanasyev I. Laser-Camera systems and algorithms to sense the environment for robotic applications. // Abstract book of Marie Curie Researchers Symposium (Warsaw, Poland), P. 16 (2011). (www.mariecurie2011.pl)
Impacts on career. The project has strong impacts on career development, helping in:
-developing the professional skills in computer vision, human-machine interaction, robotics and algorithm developing;
- getting an experience in resource, time and money management;
-supervising students, getting experience in teaching with lectures and practical exercises; -making collaboration with leading research groups and centers in computer vision and
robotics both in Italy and abroad, incl. for searching possibilities of funding; -publishing papers, participating in international conferences with oral presentations. -strengthening the researcher reputation.
The experience obtained in during the project must be valuable for future work at a research center or an industrial company.