Image De fencing Based on Binary Morphology

2019 International Conference on Artificial Intelligence and Computing Science (ICAICS 2019) ISBN: 978-1-60595-615-2

Image De-fencing Based on Binary Morphology

Meng-xiao LUO

_{, Wei-sheng XU}

_{* and You-ling YU}

School of Electronic and Information Engineering, Tongji University, Shanghai 201804, China

*Corresponding author

Keywords: Fence detection, Image inpainting, De-fencing, Binary morphology.

Abstract. Fences are widely used in our daily life, which can be seen everywhere especially in gardens, highways, zoos, etc. While the fence forms an isolation zone and provides security protection, it also brings drawbacks. For example, people can't remove the fence when taking pictures, resulting in separated scenery which reduce the aesthetic experience. Image de-fencing aims at generating images without fence, thus presenting the original appearance of the scenery. In this paper, we propose a method of fence detection based on binary morphology. By using different structural elements to obtain image information, our method can extract fences and marked them with mask. Then input the image with mask into the existing inpainting network which can generate the final de-fencing image. Our method requires neither complicated algorithm, nor time-consuming training network. It is very simple to implement, fast, and produces nearly identical results compared to more complex methods.

Introduction

Beautiful scenery can be seen everywhere. When tourists and photographers want to record the breathtaking beauty with their cameras, they may be obstructed by fences, thus leaving photos with fences. Nowadays, the safety of public places has received much attention. Although fences can protect scenery as well as buildings from damage, and guarantee people's safety, their existence affects the landscape photography. Image de-fencing process is mainly divided into two phases: 1) fence detection, and 2) image inpainting. First, the fence in the image is detected and marked to generate an image with mask which we take as an input to the next phase. Then, the existing method of inpainting is used to remove the fence to get a complete, non-occluded image.

In recent years, the research of fence detection in image processing have increased from scratch. However, the detection result is not very ideal, and the research objects are mostly fences in high-speed railway or dynamic scenes. Y. Wang [1] et al. proposed a detection algorithm of fence completeness based on vertical distribution of horizon edge, but the purpose of the algorithm is to confirm whether or not the fence is damaged or missing rather than fence extraction. S. Jonna [2] et al. used convolutional neural networks to detect fence pixels, and proposed a semi-automated de-fencing algorithm using a video of the dynamic scene, but the neural network has a narrow application because the data set for training only includes some fixed-mode mesh fences. The lack of fence types, along with simple research objects, caused the poor robustness of early methods, especially when dealing with fences of different shapes and colors. In the field of computational vision, the detection of repeated patterns in images is a hot topic, and the fence is a typical repetitive pattern. Doubek [3] et al. presented a method for image retrieval using repetitive patterns as the only feature, but the method ignored the lattice edge. We propose a fast and effective fence extraction method based on morphological operations. Combined with existing image inpainting techniques, our method can realize image de-fencing.

Proposed Methodology

to perform the set operation on the overlapping part of B and A. There are two basic morphological operations: erosion and dilation. A binary image A eroded by B is defined as (1). Dilation is the dual operation of erosion, whose definition is shown in (2).

{

}

B

AΘ = |( )_x ⊆ (1)

{

}

=

⊕B x B A

A |(ˆ)_x∩ (2)

where Bˆ represents the mapping of B to the origin, and x the pace of Bˆ . Dilation and erosion not only eliminate noise, but also find significant areas of maximum or minimum value in the image. The operation of first eroding and then dilating by using structural elements is called opening operation. On the contrary, closing operation stands for the process of letting dilation go first. Such operations constitute the basic operation of the fence detection in our paper. Opening operator is denoted as ○, and closing operator as ●.

B B A B

A =( Θ )⊕ (3)

B B A B

A• =( ⊕ )Θ (4) For different target images, preprocessing includes Top hat transformation and Black hat transformation. After the preprocessing, we binarize the grayscale image by setting threshold interval. The preprocessing and binarization affect the accuracy of fence detection to some extent. In order to make the operation process of this paper clearly, the network architecture of our model is shown in Figure 1.

Figure 1. Overview of our image de-fencing framework.

Fence Detection

Figure 2. (a), (c) are raw images. (b), (d) are masked images after operation of opening or closing.

When dealing with images which contain complex information and have similar color in its foreground and background, operations of opening and closing are not enough to obtain an ideal effect of fence detection. In such cases, top hat transformation as well as black hat transformation are required. Top hat transformation subtracts the raw image from the result image of opening, which has the characteristic of high-pass filtering. On the other hand, black hat transformation can detect the valley value in the image by using raw image to subtract the result image of closing. Fig. 3 shows an irregular fence, due to whose bending deformation, the traditional method based on texture capture received a poor effect. From Fig. 3 (b), it can be seen that the reason for the fence omission lies in the failure of setting a proper threshold value in the process of binary. Additionally, the method of fence detection based on color information is too limited to extract all the fences at one time, and the fence shown in Fig. 3(d) is a good example. Though sacrificing a certain degree of recall, we achieve a high precision and detect the fence effectively.

Figure 3. (b), (e) represent the results of top hat or black hat transformation.

For the purpose of intuitively showing the effects of our method, we compared the raw image with masked image pixel by pixel, thus coming the following table which records the information of recall and precision. In addition, according to the order of image appearance in this chapter, we named the images as following: flower, dog, sky, color.

Table 1. Recall and precision of fence detection.

Flower Dog Sky Color

Precision 0.75 0.88 0.82 0.93

Recall 0.98 1 0.78 0.94

Image Inpainting

Images, after been detected and masked, need inpainting to finish the de-fencing process. The existing image inpainting methods are mainly divided into two genres. One is traditional method, which extracts low-dimensional features by diffusion and patch. The other is the method based on deep learning, such as convolutional neural network which can predict pixel value after training.

Since the inpainting results of paper [4] and [5] are close to each other, the following figure only selects the better one to represent the effect of traditional method.

In recent years, methods of image inpainting based on deep learning have shown great potential. However, most of the networks have disadvantages of structure distortion on boundary as well as texture blur when inpaint the images. Jiahui.Yu [6] et al. proposed a feed-forward, fully convolutional neural network with contextual attention layer, which not only improves the stability and speed of the training process, but also achieves inpainting results of high quality. As shown in Fig. 4, three methods belonging to two genres are compared, providing ideas and references for future work in improvement of image inpainting.

Figure 4. (b), (e) are images inpainted by traditional method. (c), (f) by deep learning method.

Conclusion

This paper has presented a method for fence detection based on image binary morphology. By using different structural elements to detect and mask the fence, we input the marked fence into the existing image inpainting network to generate a complete image without fence. The contribution of our work is to solve the first sub-problem of image de-fencing: detect fence automatically, and we additionally compared the previous methods of image inpainting to solve the second sub-problem. Our method proved to be fast and efficient, which can achieve the effect of real-time conversion even running on a poorly configured device. Overall, our method has great potential in future work compared with the slow complex methods.

Acknowledgement

This research was financially supported by Research on Intelligent Control Algorithm of Super High-rise Complex Multi-Energy Air Conditioning System and Jiangsu Provincial Housing and Urban-Rural Construction Department - Program on the Next-Generation Smart Green Building.

References

[1] Y. Wang, Z. Yu and L. Zhu, “Automatic detection of fence completeness for high-speed railway,” Proceedings of 2011 IEEE International Conference on Service Operations, Logistics and Informatics, Beijing, 2011, pp. 585-588.

[2] S. Jonna, K.K. Nakka and R.R. Sahay, “My camera can see through fences: A deep learning approach for image de-fencing,” 2015 3rd IAPR Asian Conference on Pattern Recognition (ACPR), Kuala Lumpur, 2015, pp. 261-265.

[3] Doubek P., Matas J., Perdoch M., et al. Image Matching and Retrieval by Repetitive Patterns [J]. 2010.

[5] M. Bertalmio, A.L. Bertozzi and G. Sapiro, “Navier-stokes, fluid dynamics, and image and video inpainting,” Proceedings of the 2001 IEEE Computer Society Conference on Computer Vision and Pattern Recognition. CVPR 2001, Kauai, HI, USA, 2001, pp. I-I.