Chroma Key Approach

Initially, what was viewed to be a simplistic approach was undertaken in this project to remove the robot from the image as a novel approach to the problem. As the robot was able to be painted a solid block colour, the use of Chroma key was chosen to remove the colour range of the robot from the image. Chroma key is a widely used technique for removing backgrounds from images so that the foreground can be superimposed on another background, such as a weather map or a fictional landscape. This is achieved by filming the action scene in front of a solid blue or green screen, giving an isolated colour range that can easily be removed by simple algorithms. However, this technique, although simple, is not as well suited to the removal of moving objects in which the colour range changes in the illuminated surfaces of the object due to reflection of light, or shadowing. The challenge was to develop a method to limit the effect of the Chroma key algorithm on the image in the areas away from the object being removed from the video stream. The proposed solution to this was to use edge detection algorithms to create a region of interest around the robot and then infill that region with a block colour so that the Chroma key colour range could be reduced to removing the single colour value that had been in filled.

4.2.1 Chroma Key Implementation

To simplify the programming of this software, the OpenCV C++ library was chosen as it already has many inbuilt algorithms such as Laplace and Canny edge detection and flood fill – an algorithm that re-colours pixels relative to the original “seed” or starting

52

point of the filling operation, or relative to the neighbouring pixels depending on the parameters used when calling the flood fill algorithm.

Initially the raw chroma key algorithm was implemented to experiment with the colour ranges required to remove the robot. This also demonstrated the flaw in using this type of algorithm for foreground non-stationary object removal. Figure 4.1 shows an example of the differing colour ranges as the illumination changes when the robot moved.

Figure 4.1 - Demonstration of altered colour range as object moves

With a correctly functioning implementation of chroma key achieved, the next step was to implement the edge detection algorithm which would be later used to create the region of interest for the colour infill. Laplace edge detection was selected as the initial algorithm for edge detection to create the region of interest that would restrict the chroma key algorithm. Laplace edge detection produces good definition of the object outlines, without requiring intensive processing power. Figure 4.2 shows the implementation of Laplace edge detection.

53

Figure 4.2 - Implementation of Laplace edge detection

To implement the Floodfill algorithm that is included in the OpenCV library, an iterative approach was adopted to search through the image to find groups of pixels within the correct colour range, and then fill the region containing this group of pixels with the pre-determined block colour for the chroma key to filter out. The Laplace output image was used as a mask for the Floodfill algorithm which was implemented such that it will not fill across non-zero pixels in the mask (i.e. the object edges), keeping the operation within the required region. A demonstration of the Floodfill algorithm implemented on an image processed with Laplace edge detection is shown in Figure 4.3.

54

Now that there was a set block colour within the bounded region of the robot, it was simple to use the chroma key algorithm to remove the set RGB colour (in this case 255, 0, 0 was used for the RGB block colour). Figure 4.4 shows the both the implementation of the chroma key algorithm with the use of Laplace edge detection, and without the use of edge detection for comparison.

Figure 4.4 - A demonstration of chroma key applied with edge detection implementation (left) and without edge detection implementation (right)

As shown above, the implementation of the edge detection and chroma key algorithms combined improved the outcome for removing dynamic foreground objects, however, the result shown above still needed further improvement to be suitable for use in the project.

4.2.2 Chroma Key Discussion

The major weakness of this approach has proven to be the strength of the Laplace edge detection. As the video image from the webcam was low resolution, with relatively high noise, Laplace edge detection proved to be less effective for this application due to its sensitivity to noise [110] without strong definition between the object colour and its surroundings due to the noise in the image, the outlines of the edges produced by the Laplace algorithm become weak, with small (sometimes pixel sized) breaks in the edges

55

allowing the Floodfill algorithm to “spill out” and fill larger areas of the screen. This was unacceptable for the operation of the virtual mirror system, and therefore the approach for removing the robot from the video stream required alteration. The Laplace edge detection algorithm was replaced Canny edge detection, which has better response to noisy images than Laplace edge detection, but is more computationally complex. The Canny edge detection improved the quality of the image output, producing more defined edges around the robot, but on visual examination of the real-time video stream, the added computational complexity had a detrimental effect on the response time of the video stream, producing a jittery video image.

The use of a controlled environment helped to improve the quality of the robot removal algorithms, by using a contrasting background to the robot, and adjusting lighting to minimise reflections/shadowing. However, the video image was still unstable, and further investigation was required.