To improve the identification of different handles and to increase the friction with the arm end effector, a new handle design with flat panels on the upper and front sides has been resolved. The upper flat panel is used for fixing different colored or pictorial marks (glyph, 2D barcode, etc.) to distinguish multiple handles as shown in Fig. 6.10.
6: Visual Labware Manipulation
The H20 mobile robot with the Kinect sensor V2 are used to perform the grasping and placing tasks for the required handle where the desired labware is positioned. The tasks of handle manipulation are based on the following scheme: in the 1st step, the robot checks around the workspace to detect the desired handle which has to be transported to the required lab. In the 2nd step, the position of the target related to the arm shoulder is calculated. In the 3rd step, the robot uses the position information obtained from the 2nd step to insert it into the arm kinematic model. In the 4th step, the arm moves to the required position and grasps the handle. After finishing the grasping task successfully, the robot moves to the desired destination to place the handle on the required holder position. The HSV color system has been used to detect the required handle where the RGB color system can easily be affected by changes of lighting conditions. HSV system can be considered the most powerful system to be used for color segmentation because it is more robust to control the brightness. To differentiate multiple handles, different color can be used. In case of existing more than one object in the view that has the same color, then it will be crucial to use additional data sources, such as area, or shape to find the objects of our interest. HSV color segmentation method with shape and area information was used with Kinect V2 to detect the required handle as shown in Fig. 6.11. The flat panel of the handle has a red color with a rectangle shape and the area of the detected object is checked whether it is within the required range or not. Thus, it can be guaranteed that the detected object is related to the flat panel of the required handle. Histogram equalization is applied to the image to increase its global contrast as a preprocessing step before the HSV color segmentation. At the end, a polygon with cross is drawn around the target to define it and to identify its center point. For the transportation tasks, the information about the required handle, grasping station, and placing station are sent to the H20 mobile robot. The required labware, which is needed to be transported, has to be placed on the labware container attached with the target handle. Furthermore, a pictorial mark with specific textures and features can also be fixed on the handle to be recognized by Kinect V2 using SURF algorithm as shown in Fig. 6.12 [199]. SURF can be considered as an efficient object recognition algorithm with rotation- and scale-invariant descriptors and detectors. The process of recognition is separated into an offline object modelling stage and an online recognition stage. The objects’ images are saved in the database in the offline stage. Given a new test image, the online local descriptors are matched with the stored models in the database to perform the identification process.
6: Visual Labware Manipulation
Figure 6.11: The recognition of handle using HSV and shape detection.
Figure 6.12: The recognition of handle using SURF.
The recognition strategies are also applied to detect the holder for the placing tasks. Color, shape, and area information can be used to detect the required position for labware placing. Also, some pictorial marks can be fixed on the holder to be recognized by the Kinect V2 as shown in Fig. 6.13 and Fig. 6.14 respectively.
Figure 6.13: The recognition of holder using HSV and shape detection.
6: Visual Labware Manipulation
After the target recognition, the coordinates of the center point of this target in the image frame has to be found. Then, the position of this center point related to the Kinect is identified using a mapping process. The required point is mapped from the color frame to the depth frame of Kinect. Then, another step is performed to map the point from depth frame to the Kinect space coordinates. The position of the center point is used as reference for calculating the grasping or placing point positions which the arm end effector has to reach. After identification of the position of the required point related to the Kinect, an extrinsic calibration is applied to transform the position information to be related to the arms’ shoulders. Then, the inverse kinematic model is used to move the arm to the target and perform the task [199]. Fig. 6.15 shows the center and the grasping points of the handle and how the arm reaches it.
Figure 6.15: The grasping point and how the arm reaches it.
For the labware transportation framework with the H20 mobile robot, the system consists of three main coding platforms which are the robot navigation control (RNC) [201][202], the robot arm control (RAC), and the Kinect control (KC) for object detection and localization. It is a complex issue to integrate the three main platforms in a single coding project where each one includes a huge number of coding lines. Also, this integration is problematic because it increases the possibility of coding bugs which affect the whole system. To cope with this issue and to realize an intelligent performance, it is required to separate the coding platforms and develop a communication system. The client-server communication model (asynchronous socket) can enable the control system of each platform to interact with the others over Ethernet. The client-server connection is realized through a specific socket address using TCP/IP where the socket address is a combination of IP address and a port number. For the task of labware transportation, two client-server models have been developed. The first model is to connect RNC with the RAC. The second model is to connect the RAC with the KC where the RAC has a client and a server platform at the same time as shown in Fig. 6.16. The RNC (client) sends the orders to the RAC (server) that include the required task (grasping or placing) with other information related to the target specifications. The RAC (client) sends the target specifications
6: Visual Labware Manipulation
to the KC (server) to be found. Then, the KC sends back the target position to the RAC which in turn inserts the received information to the kinematic model to control the robotic arm. In case that the target is not existing in the Kinect view or it is not possible to manipulate it, a feedback information is sent to the navigation control related to the next procedure [199]. This feedback information includes several decisions such as changing the robot position or performing the next task. The changing of robot position helps to get better view for the Kinect sensor and to make the target within the arm workspace.
Figure 6.16: The architecture of client-server model for handle manipulation.
The grasping and placing tests were performed by placing the robot at different positions in front of the workstation. Prior to each grasping or placing attempt, the robot is placed in a new arbitrary position. The robot has a positioning error in the range of ±2cm in X-axis (left/right of the workstation) and ±3cm in Y-axis (distance) in front of the workbench. As can be seen in Table 6.1, the results show the overall success rate for grasping and placing which confirms that the statement of the used methods are acceptable enough for labware manipulation and transportation [199]. The entire required time for performing the task is about 65 seconds including the recognition, localization, IK calculation, and arm movement. The flowchart of arm manipulation system is shown in Fig. 6.17. According to the arm’s workspace and to the robot position in front of the workstation, each arm can reach 2 labwares which are alongside each other. The work has been developed using Microsoft Visual Studio 2015 with C# programming language. The project is running on a Windows 10 platform in the H20 tablet.
Table 6.1: The handle manipulation tests.
Methods Attempts Successful grasp Successful place
HSV filtering 50 92% 90%
6: Visual Labware Manipulation
Figure 6.17: The flowchart of arm manipulation system.
A 12V-18A/h battery with voltage stabilizer have been installed on the robot body to provide the Kinect sensor with the required power during the transportation tasks as shown in Fig. 6.18. Some connectors are fixed with the power cable of Kinect to give the possibility of using the AC and DC power for Kinect at the same time [200].
Figure 6.18: The battery and voltage stabilizer for Kinect sensor.