The Implemented Graphical User Interfaces

In order to provide the user with the ease of using the stereo camera system, it is crucial to have a graphical user interface (GUI) which can represent the information and actions efficiently. A graphical user interface is a program interface item that allows the user to interact with the programs in more ways than just typing commands. It offers graphical icons and visual indicators to represent the information and action efficiently. A well-designed interface can help the user to use the program without the necessity to understand the theories and complicated algorithms.

Three major GUIs have been implemented during the study of this research. All of these interfaces were developed based on the approach of developing applications with HALCON as described in Section 3.3.2. The first interface was implemented in Visual basic as an initial interface for the torsion test. This GUI provides the user with the basic functions to calibrate the stereo camera system and save the image data during testing. The calibration can be performed by processing these calibration image data, extracting the calibration circular marks, and creating the calibration files and saving them to the hard disk of the computer. This was done with the help of functionalities provided by the HALCON software. Besides the functions of loading and saving images, the GUI also provides an interactive initial parameter setting which allows the user to set the appropriate parameters of the stereo camera system.

The second GUI was improved by being implemented in MFC, which includes a variety of controls. This GUI is developed in MFC dialog-based application using Visual C++ 2010. It was implemented to be used for the torsion test when using two stereo camera systems. Besides the ability to connect and view up to 4 cameras, the second GUI can also be used to calibrate the stereo camera system in real time or later after saving the calibration images. Figure 3-11 shows several controls employed in the system. The two top windows can view the stereo images of the first binocular stereo system while the two bottom windows can view the stereo images of the second binocular stereo system.

70

As shown in Figure 3-11, the GUI includes in general three control panels named as ‘Image Acquisition Devices’, ‘Calibration Control’, ‘ and Recording’. The first panel contains control buttons for performing different tasks. The browser button is used to select the folder where the acquired images are saved to. The other buttons can be used to carry out different tasks such as connecting or disconnecting the camera system. The cameras connected to the computer can be detected and then selected from the combo box to be used for the stereo imaging system. The second panel is used to perform the calibration of the stereo camera system. The number of images to be used for stereo camera calibration is required to be entered. The description file containing world measurements of the used HALCON calibration plate is also required to be selected. This panel also includes control for camera parameters. The combo box is designed to let the user select the camera model of the system. The focal length of the lenses can be entered in the associated box.

Moreover, the width and height of the sensor cells can be adjusted in the associated box. Furthermore, the parameters used to extract the marks of the calibration plate can be specified in the given boxes. The third panel of this GUI is used to start saving the image data during testing based on a time interval that can be adjusted according to the desired time.

Finally, a third GUI was developed in MFC and used during the compression tests, as shown in Figure 3-12. The interface was used to calibrate the stereo camera system and to record the image data during testing. The calibration of the stereo camera system with this GUI was further improved by allowing the calibration to be performed in real time or later after saving the calibration images. Another helpful feature of the GUI is the capability of automatically determining the presence of the calibration plate of HALCON in the image sequences, allowing the user to locate the plate easily and enhance the speed of calibrating the stereo camera system. Additionally, the extracted calibration parameters of both cameras can be viewed in the interface as shown in the above-mentioned figure.

71

Figure 3-11: Developed graphical user interface used during the torsion tests.

Figure 3-12: Developed graphical user interface used during the compression tests.

Window 2 Window 1

Window 3 Window 4

72 3.6.4 The Implemented Algorithms

3.6.4.1 Camera Calibration Algorithm

An algorithm was developed to calibrate the binocular stereo camera system based on the image processing software, HALCON. A part of the calibration code implemented in MFC is shown in Figure 3-13. The calibration algorithm consists of several steps. The basic steps of the calibration process are shown in Figure 3-14. First, the calibration images are acquired, saved and processed by HALCON operators in a loop. The find-caltab operator, for example, is used to find the region of the calibration plate in the processed image and find-marks-and-pose determines the 2D position of the calibration marks and estimates the pose of the plate. With this information, the operator binocular-calibration determines the camera parameters of the binocular camera system.

Figure 3-133: A part of the implemented C++ calibration code for calibrating the stereo camera system.

73

Figure 3-144: A diagram of the basic steps for stereo camera calibration.

Left Images Right Images

Input Initial Parameters of the cameras

Apply find-caltab and find-marks-and-pose operators

Pose measurement of the calibration plate

Stereo camera parameters

 Parameters of the left camera

 Parameters of the right camera

 Relative camera pose

Stereo camera calibration

74 3.6.4.2 Deformation Measurements Algorithm

Since the usage of cameras as measurement devices, different algorithms have been implemented for measuring the 3D coordinates of the photogrammetric targets.

Circular-shaped targets are easy to detect and they have been used in different applications to achieve the highest accuracy (Clarke, 1994,Luhmann, 2010, Otepka and Fraser, 2004). All of the 3D deformation measurement algorithms were written in HALCON which provide a powerful library for processing the stereo images.

The implemented algorithms process the stereo images, detect the target points, and determine the 3D coordinates of the points. The whole process was programmed in the software to be automated. Figure 3-15 summarizes the concepts and steps involved in the algorithms. The process starts with reading the calibration files, recorded during the calibration process, containing the internal and external parameters of the binocular stereo system. The stereo images of the targets recorded by the left and right cameras during testing were then displayed to be used in the image recognition process. The sub pixel-precise segmentation method provided by the software was used to reduce the area of interest of the stereo images by using the threshold values of the pixels corresponding to the targets. The potential targets that meet a specific shape and size were then extracted with sub-pixel accuracy.

These targets were fitted to ellipse shapes and their image coordinates in both the left and right images were measured. The stereo-matching problem was then solved by sorting these targets in each of the left and right images according to their order in row and column coordinates from top to bottom. They were labelled and numbered to show how they match each other. The 3D coordinates of the target points were then determined from the intersection of two lines of sight within the binocular camera system using the theory of triangulation (Hartley and Zisserman, 2003). This was achieved by applying the operator ‘intersect_lines_of_sight’ to the extracted target points. The 3D measurements were determined relative to the coordinate system of the left camera. To define a world coordinate system, three target points were employed. The first point defines the origin of the coordinate system, the second point lies on the x-axis and the third point lies on the y-axis. The transformation matrix was then determined from these points. Using the transformation matrix together with the calibration parameters of the stereo camera

75

system, the determined point coordinates were transformed from the coordinate system of the first camera into the world coordinate system. These 3D coordinates were then saved to the project computer and used in the calculation of the displacements and rotations of the targets during the period of tests.

Figure 3-155: Flowchart of the stereo reconstruction algorithm based on HALCON.

Stop to examine the images Detection

successful

Process the stereo images and find the conjugate points

Yes

No Use the determined stereo camera calibration

Use intersect_lines_of_sight to determine the 3D coordinates

Load the stereo images of the binocular system

Detect the circular targets and mark them

Stop Start

76

4. VALIDATION AND EVALUATION OF THE PROPOSED