Automatic Extraction Device Nameplate Area from Digital Image

2018 2nd International Conference on Modeling, Simulation and Optimization Technologies and Applications (MSOTA 2018) ISBN: 978-1-60595-594-0

Automatic Extraction Device Nameplate Area from Digital Image

Lin LI

, Xiao-bo LIU

and Yan-di LI

Wuhan NARI Limited Liability Company of State Grid Power Research Institute

3

Department of Computer Science and Technology, China University of Geosciences, China

Keywords: Machine vision, Convolution kernel, Morphology filter, Object recognition.

Abstract. Since that nameplate recognition plays an important role in intelligent manufacturing，we explored an image processing solution based on the digital image of the metal nameplate of power equipment. At first, the gray image and threshold segmentation are used to realize the image preprocessing. After this, the application of the three edge detection operators of the scharr operator, the sobel operator and the canny operator in the actual image is compared to realize the image edge detection process. Based on the morphological method, the noise elimination effects of three different structural elements and convolution kernel are analyzed afterwards. The method of identifying the nameplate area in steps is designed. Finally, through the contour detection and perspective projection, the image processing results are improved, and the extraction process of the complete digital image nameplate area in complex environment is realized.

Introduction

At present, intelligent identification gradually penetrates into many aspects of life. Intelligent equipment maintenance schemes have been proposed for power systems. Instead of human eyes, an intelligent identification system for device nameplate uses visual sensors to acquire environmental information and convert it into a data matrix. Also, such a system replaces human brain with computer. Software is used for image processing and analyzing to extract the nameplate content, thereby achieving unified collection and management of device information[1]. Therefore, image capture and object recognition in nameplates of power or other equipment has become a focus.

Object recognition, the basic steps of image analysis, pattern recognition, and computer vision[2,3], is an important research direction in computer vision. In an environment with complex objective, the accuracy of object recognition may be greatly reduced by a large amount of interference information. Consequently, results are not always satisfactory. Therefore, for different recognition objects in complex background, different image processing processes are designed to eliminate the noise in the objective environment. In this way, result of object recognition becomes more accurate than before. Then, a good foundation is laid for the next step. The main operations in object recognition include image acquisition, image preprocessing, feature extraction, detection segmentation and advanced processing.

We study the object recognition process of extracting the device nameplate area for the metal reflective nameplate. In this paper, we explore the different apply effect of the image processing strategy, image processing and other image processing strategies in the image denoising process under the complex shooting environment based on different nameplate material to find more adaptability and robust nameplate area extraction strategy. At the same time, we carry out in-depth mining on image processing methods in complex backgrounds.

Background

Thresholding. Threshold segmentation, one of the common image segmentation techniques, also known as image binarization, classifies pixels according to different eigenvalues. Such a kind of methods are simple and stable and can reduce the computational complexity. Animage processed by threshold segmentation has only two pixels’ values of 0 and 1.Thus, the subsequent features of such an image can be identified based on the statistical characteristics of the pixel values. Threshold segmentation methods are usually divided into global methods and local methods. In this paper, global methods are selected based on the actual shooting environment of the device nameplate. A thresholding operation can reduce the amount of data for the denoised image and provide a foundation for further processes such as contour finding. In Fig.1, the results of different threshold segmentation methods are shown.

Figure 1. Example of results for different threshold segmentation methods.

Edge detection. Edge detection is a mathematical method of identifying points in a digital image whose luminance and gray values are discontinuous or abruptly changing. A curve segment formed by a set of points with sharp change in gray level is called an edge. Such discontinuous curve segments can usually be detected by derivation. In general, first-order and second-order derivatives are used. The first derivative usually refers to the first derivative expression. For example, the gradient value corresponds to the first derivative of the edge and can be used to calculate the partial derivative component in each direction of the local region, thereby finding the edge in the local direction. The same goes for the second derivative. Sobel algorithm and Canny algorithm are selected as the representative for first derivative edge detection and the second derivative edge detection, respectively.

Figure 2. Example of edge detection for different operators.

Morphological operations. Mathematical morphology introduces the topology and geometric concepts in continuous space and discrete space, such as size, shape, convexity, connectivity and geodesic distance. Morphology is originally developed for binary images and then extended to half-color images. Morphology operations, which convert images according to the above functions, are the foundation for processing morphological images. The morphological operations which are mainly applied to collection of pixel points through structural elements include erosion, expansion, opening and closing. These operations can be combined to get composite algorithms.

Let A be a collection of pixel points and B be a structural element. In an operation, B acts on A.The operator for expressing expansion, “⊕”, corresponds to the operation “XOR” in mathematics. Thus, A expansion with B can be recorded as A ⊕ B and defined as:

A⊕B =｛x | [ (B̂)X ∩ A ] ≠ ∅｝. (1) “ ”, the operator for expressing erosion, also known as corrosion, corresponds to the operation "same or" in mathematics. Thus, the erosion of A with B can be written as A B and defined as:

A B =｛x | ( B )X ⊆ A｝. (2) The process of erosion and re-expansion is called open and expressed by “ ”. Thus, the opening of A with B can be written as A B and defined as:

A B = (A B) ⊕ B. (3) The process of corrosion after expansion is called closing and expressed by “•”. Thus, the closing of A with B can be written as A•B and defined as:

Our Work

By investigating the effects of different image processing steps such as image preprocessing, morphological filtering and color filling in the image denoising process, a device nameplate region extraction scheme is proposed by us for metal reflective nameplates. The flow of our scheme is shown in Fig. 3.1. To judge whether the results can meet requirement, the results are evaluated by different image processing strategies with multiple combinations of processing order and parameter value.

Figure 3. Flow chart of equipment nameplate object identification method.

Image Preprocessing. For the initial image (Fig. 4), in order to simplify the complexity of the RGB image color distance calculation, and prepare for the subsequent operation, grayscale processing is performed to obtain a single-channel grayscale image. The pixel color value is represented by the gray level.

As shown in Fig. 5, After grayscale transformation, in order to capture the prominent features, texture and material, of nameplate object, Scharr operator (specified ksize = -1) is used to construct gradient amplitude representation of grayscale image both in the horizontal and vertical directions. Through the gradient transformation in Eq.3.1, the output image area will have a high horizontal gradient and a low vertical gradient.

Figure 4.The original image I. Figure 5. Scharr operator rendering on Fig.4.

Next, the image is initially noise-reduced with a low-pass smoothing filter and a 5*5 smoothing matrix. Frequency domain of the image is reduced by replacing the value of each pixel with the mean of the value of pixels in its adjacent rectangle. Then, intense intensity changes more gradual. A filtered image with a noticeable smoothing effect is obtained as shown in Fig.6.

It can be seen that the attempt for reducing noise on the grayscale image shows no obvious effect. Further, such an operation costs too much since each pixel is involved in computation. Therefore, we try to process the image with a thresholding method, the maximum inter-class variance method. In this method, the threshold value for distinguishing the foreground and background of the image, T, the ratio of the number of pixels in the background to the total number of pixels in the image,𝑝₀, the proportion of the number of foreground pixels to the number of total pixel,𝑝₁ the average gray values of the background 𝑞₀and the foreground 𝑞₁ need be set. Here, the average gray value of the full image is Eq.6:

average = 𝑝0∗ 𝑞0+ 𝑝1∗ 𝑞1. (6)

The variance between the foreground and the background is expressed as Eq.7.

variance = 𝑝₀∗ (𝑞₀ − 𝑎𝑣𝑒𝑟𝑎𝑔𝑒)2_{+ 𝑝}

1∗ (𝑞1− 𝑎𝑣𝑒𝑟𝑎𝑔𝑒)2 (7)

As Eq.8, we can find the segmentation threshold when the class variance is maximum.

Thre = 𝑝₀∗ (𝑞₀ − 𝑞₁)2_{. (8)}

Our threshold selection method classifies pixel points by feature values. As shown in Eq.9, values higher than the threshold are changed to the maximum, while values lower or equal to the threshold is set to zero. In this way, image binarization is completed to reduce noise.

dst(x, y) = {𝑚𝑎𝑥𝑉𝑎𝑙𝑖𝑓𝑠𝑟𝑐(𝑥, 𝑦) > 𝑡ℎ𝑟𝑒𝑠ℎ

0 𝑜𝑡ℎ𝑒𝑟𝑤𝑖𝑠𝑒 . (9)

Figure 6. Smoothing effect on Fig.5. Figure 7. Threshold operation effect on Fig.5.

we use a small convolution kernel method to eliminate isolated noise points based on the threshold image.

In convolution, a suitable filter matrix needs be selected. Then, the pixel matrix composed of each pixel and surrounding pixels with the selected matrix are both multiplied. The value of multiplied pixels is produced. Here, the filter matrix is called convolution kernel. The effect of convolution filter is in Fig.8.

Figure 8. Convolution operation effect I on Fig.7.

The convolution result can be further adjusted by changing the size of the convolution kernel. As shown in Fig.9, the independent small noise points in the edge portion are basically eliminated, while the effect of eliminating small noise points is more obvious after replacing the processing object.

Figure 9. Convolution operation effect II.

Morphological filtering. After the operations in the previous subsection, the noise reduction process is basically completed. In order to highlight the shape of the object, the pattern gap caused by the metal highlight of the nameplate needs be filled. Meanwhile, the protruding block that affects the result needs be filtered out. To these ends, we use the morphological operations of corrosion, expansion, opening and closure to perform the composite morphology algorithm processing.

In morphological operation, let input image be F, the structural elements be collectively called K and E (x, y) represents the edge of image. The composite morphology algorithm is designed as follows:

E ( x , y ) = [ ( F • K ) K ] ⊕ K. (10) that is,

E ( x , y ) = { [ ( F⊕K ) K ] K } ⊕ K. (11) At this time, the gradient operator of the edge detection of each step of morphological operation is given as follows:

GD2 ( x , y ) = f ( x , y ) K

GD3 ( x , y ) = f ( x , y ) ⊕K

GD ( x , y ) = GD1 ( x , y ) - GD2 ( x , y ) + GD3 ( x , y )

= [ f ( x , y )⊕K–f ( x , y ) K ] – [ f ( x , y ) K ] + [ f ( x , y )⊕K ]. (12) In the operation process, it is found that in the environment with complex and noisy interference, the result of the operation of a single structural element is often unsatisfactory, so we try to improve the algorithm. Three structural elements differing in scale and shape are selected and applied, for adding different processing functions through different properties of the structural elements. The morphological operation expression after compounding is:

E ( x , y ) = [ ( F • K1 ) K2]⊕K3

= { [ ( F⊕K1) K1 ] K2 }⊕K3. (13) Then the gradient operator of the edge detection of the morphological operation is as follows:

GD (x, y ) = GD1 ( x , y ) - GD2 ( x , y ) + GD3 ( x , y )

= [ f ( x , y )⊕K1–f ( x , y ) K1 ] – [ f ( x , y ) K2 ] + [ f ( x , y )⊕K3 ]. (14) The matching effect after compounding is shown in Fig.10.

Figure 10. Single & Multiple structural element operation results on Fig.8.

Also, the suitable selection of structural element scales is also beneficial to improve noise reduction efficiency [4]. The zero square matrix is selected as the structural element, which avoids the influence of the value of the structural element on the image to aggravate the noise.

Figure 11. Morphological operation result.

Contour Extraction. At present, we have completed the noise reduction processing and edge detection on the nameplate area, and obtained an area that is extremely close to the shape of the object in the original image. Then, we can use the contour search algorithm to find the largest rectangular frame surrounding the object to mark the object area. We use the corner detection method todetect the four edge points of the object, as in Fig.12.

Figure 12. Contour extraction example

Further, the perspective projection operation is performed so that the irregular quadrilateral can be converted into a rectangle with square corners at right angles. Thus, as is shown in Fig.13, it is convenient for inspection of the device administrator and further operations.

Figure 13. Perspective projection on fig.12.

Summary

elements are designed based on morphology. Further, the small kernel convolution is used respectively during the noise reduction process. After that, both contour detection and perspective projection are used to improve the image processing result. Finally, the extraction process of the complete digital image nameplate area in complex environment is realized based on the above steps.

In our work, the main problem needing be solved is the special texture of the object image and the noise interference caused by the environment in which it is located. After a series of attempts, the technical route was continuously revised and improved. When the regular object identification method was used for the device nameplate, the effect was not satisfactory due to the special material of the nameplate and the environment. In conclusion, the appropriate image processing algorithm should be analyzed and designed based on characteristics of image and object to obtain a good recognition effect.

References

[1] Liu, Y., Cheng, M. M., Hu, X., Wang, K., & Bai, X. (2017, July). Richer convolutional features for edge detection. In Computer Vision and Pattern Recognition (CVPR), 2017 IEEE Conference on (pp. 5872-5881). IEEE.

[2] Steger, C., Ulrich, M., & Wiedemann, C. (2018). Machine vision algorithms and applications. John Wiley & Sons.

[3] Joshi, M. A. (2018). Digital image processing: An algorithmic approach. PHI Learning Pvt. Ltd.

[4] Howard, A. G., Zhu, M., Chen, B., Kalenichenko, D., Wang, W., Weyand, T., ... & Adam, H. (2017). Mobilenets: Efficient convolutional neural networks for mobile vision applications. arXiv preprint arXiv:1704.04861.

[5] Laganiere, R. (2017). OpenCV 3 Computer Vision Application Programming Cookbook. Packt Publishing Ltd.

[6] Vernieuwe, H., De Baets, B., & Verhoest, N. (2018). A mathematical morphology approach to the identification of drought events in space and time.

[7] Li, Y., Zuo, M. J., Chen, Y., & Feng, K. (2018). An enhanced morphology gradient product filter for bearing fault detection. Mechanical Systems and Signal Processing, 109, 166-184.

[8] Chang, C. H., Lai, W. S., & Chuang, Y. Y. (2018, October). Generating a Perspective Image from a Panoramic Image by the Swung-to-Cylinder Projection. In 2018 25th IEEE International Conference on Image Processing (ICIP) (pp. 3179-3183). IEEE.

[9] Klekiel, T. (2017). Application of the Fundamental Solution Method to object recognition in the pictures. Image Processing & Communications, 22(3), 13-22.

[10] Perner, P. (2017, June). Verification of Hypotheses Generated by Case-Based Reasoning Object Matching. In International Workshop on Combinatorial Image Analysis (pp. 66-78). Springer, Cham.

[11] Perner, P., Jänichen, S., & Perner, H. (2006). Case-based object recognition for airborne fungi recognition. Artificial Intelligence in Medicine, 36(2), 137-157.

[12] Wu, H., Bie, R., Guo, J., Meng, X., & Zhang, C. (2017). CNN refinement based object recognition through optimized segmentation. Optik-International Journal for Light and Electron Optics, 150, 76-82.

[13] Song, R., Zhang, Z., & Liu, H. (2017). Edge connection based Canny edge detection algorithm. Pattern Recognition and Image Analysis, 27(4), 740-747.

Extracting Out-of-Gauge Goods Contour. In Applied Mechanics and Materials (Vol. 775, pp. 404-408). Trans Tech Publications.

[15] Ying-Dong, Q., Cheng-Song, C., San-Ben, C., & Jin-Quan, L. (2005). A fast subpixel edge detection method using Sobel–Zernike moments operator. Image and Vision Computing, 23(1), 11-17.

[16] Khairuzzaman, A. K. M., & Chaudhury, S. (2017). Multilevel thresholding using grey wolf optimizer for image segmentation. Expert Systems with Applications, 86, 64-76.

[17] Maolood, I. Y., Al-Salhi, Y. E. A., & Lu, S. (2018). Thresholding for medical image segmentation for cancer using fuzzy entropy with level set algorithm. Open Medicine, 13(1), 374-383.

[18] Khashman, A., Oyedotun, O., & Sadikoglu, F. (2016). Intelligent recognition of chelonioidea sea turtles. Procedia Computer Science, 102, 617-622.

[19] Gonzalez Rafael, C. (2008). Digital Image Processing, /Rafael C. Gonzalez, Richard E. Woods.