Panel Lights Detection Based Automatically Equipment Monitoring Through Deep Learning Method

2019 International Conference on Computation and Information Science (ICCIS 2019) ISBN: 978-1-60595-644-2

Panel Lights Detection Based Automatically

Equipment Monitoring Through Deep

Learning Method

Xin Wang, Yue Jiang, Fan Yang, Yu Zeng, Qi Zhang,

Huaihao Wei, JiaZhou Li, Dou Wu, Yuqiang Fan

and Yaoran Huo

ABSTRACT

Automatically equipment monitoring is important in automation field. It asks for the combination of image processing, pattern recognition and the corresponding computer vision technology. Since most equipment status could be read through the panel lights, this paper proposed a deep learning method to automatically monitor devices through detecting their signal lights from panel images captured by digital camera. The entire method employed Unet model to be the backbone model. The feature maps of different scales extracted by the encoder of Unet were weighted fused together to generate the final output of the detected lights. The model were trained on our equipment panel image set. The validation showed that the proposed method could detect the small panel lights accurately.

1. INTRODUCTION

The automatic detection of equipment status is one of the important problems to be solved in the field of engineering application. Based on computer vision and modern image processing technology, the automatic detection and response of equipment running state is essentially the detection of signal light status on the image or video. The signal light of an equipment is used to reflect the working status

Xin Wang, Yu Zeng, Huaihao Wei, Jiazhou Li, Dou Wu and Yuqiang Fan, Information and Communication Company, State Grid Sichuan Electric Power Company. Chengdu, 610041, China

and alarm status of the equipment. Recognizing signal light status automatically can greatly reduce the labor consuming, improve the efficiency, and avoid misunderstanding caused by human factors such as fatigue and carelessness. Due to that, it has great application prospects in the field of automatic monitoring and driverless driving.

Recognizing signal light status means to correctly locate and identify the status of the indicator light from the image. There were a lot of researches focusing on this issue, and some of them had made certain progress and achievements. The current research methods could be divided into color based methods, template matching methods and machine learning methods. Based on the color characteristics, the threshold value was used to segment the target area of interest [1-8]. For example, statistical color histogram and other methods were used to determine the threshold range of the signal light in a certain color space, and then the image was segmented in the corresponding color space. These methods were too sensitive to resist the change of illumination and thus not robust. The methods based on template matching [9-10] compared the similarity between the empirical template and the target region to detect and recognize targets. The methods were simple and real time, but not suitable for complex background. In recent years, the deep learning methods were successfully applied in detection and identification [11-15].

At present, the main difficulties in signal light detection and recognition are as follows: 1. the signal detection mission is a typical small target detection. Normal feature extraction method depends on the statistical information, which would be less accurate if the target is too small to have enough pixels. 2. The targets are densely distributed in the space. For example, the equipment are always located in the cabinet which has limited space, which means the lights are close to each other and hard to be recognized. 3. The extracted features are affected by the environment. The color and brightness of the image will keep change in different illumination. Furthermore, reflection, refraction, diffraction and other physical phenomena will cause halo and chromatic aberration. Due to the above reasons, the detection rate of the current method is not high enough to be directly applied signal light detection applications.

Figure 1. Example of device panel. The purple box indicate the signal lights to detect.

2. MATERIALS AND METHODS

2.1 Workflow of Device Status Detection

The workflow of the proposed device status detection system was shown in Figure 2, including image acquisition, preprocessing and signal light detection. In this paper, the images were collected under different illumination conditions through digital camera. The image preprocessing operation aimed to keep consistence of the image mean and variance through color normalization. The normalized image would enter the improved Unet detection model for signal detection.

Image Acquisition

Image Preprocessing

Signal Lights Detection

Figure 2. Flow chart for signal light detection.

2.2 Improved Unet Model

small signal light by directly using the original Unet model. Considering that for smaller targets, the amount of information that can be provided by the feature maps from finer scale is more than that provide by the feature maps from coarser scale. This paper proposed a weighted concatenation step before combining the feature maps from different scales. We added weighting factors to the feature maps of each scale _i, i∈[1,4]. _i were trainable parameters. When concatenating the feature

maps from two scales, we first use the _i to penalize the finer scale feature map and concatenate it with the coarser scale feature map.

Figure 3. Improved Unet model.

2.3 Loss Function

In supervised learning, the objective function (loss function) is used to evaluate the degree of consistency between the model output estimation and the corresponding labels. In this paper, we used the weighted sum of cross entropy and dice coefficient as the loss function to train our model.

2.4 Data Set

(a) Device panel image (b) manual label image Figure 4. Image example. (a)device panel image; (b)manual label image.

2.5 Quantitative indicators

We conducted statistical analysis to evaluate the performance of the proposed algorithm. Intersection over Union (IOU), recall and precision were calculated following Eq. (2). Here, TP, TN, FP and FN indicated true positives, true negatives, false positives and false negatives, respectively.

{

𝐼𝑜𝑈 =_{𝐹𝑃+𝑇𝑃+𝐹𝑁}𝑇𝑃

𝑅𝑒𝑐𝑎𝑙𝑙 =_{𝑇𝑃+𝐹𝑁}𝑇𝑃

𝑃𝑟𝑒𝑐𝑖𝑠𝑖𝑜𝑛 =_{𝑇𝑃+𝐹𝑃}𝑇𝑃

(2)

3. RESULTS

(a) (b)

(c) (d)

Figure 5. Detection examples of the proposed algorithm.

The quantitative metrics of the proposed improved Unet model and original Unet model on signal lights detection was shown in TABLE I.

TABLE I. METHOD PERFORMANCE.

IoU Recall Precision

As can be seen from TABLE I, in terms of IOU, compared with the original Unet model, the improved Unet model in this paper is improved by 4%. At the same time, the recall rate and precision of the improved model are also improved.

4. CONCLUSIONS

In this paper, aiming at the problem of automatic equipment state detection based on computer vision and image processing technology, we proposed a signal light detection algorithm based on improved Unet model. It could be use to realize the accurate detection of signal lights on device panel images. And with its help, it was possible to monitor the devices automatically. In this paper, the weighting fusion of the original Unet model on the three scales was proposed, and the weight of each scale was automatically updated during the training process to improve the detection performance of the model for small targets. Experiments showed that under the same conditions, the improved Unet model was better than the original Unet model in the IOU, recall rate and precision.

REFERENCES

1. Soares J C D S, Borchartt T B, Paiva A C D, et al. 2018, “Methodology Based on Texture, Color and Shape Features For Traffic Light Detection and Recognition.” 2018:1-7.

2. Jie Y , Xiaomin C , Pengfei G , et al. 2013. “A new traffic light detection and recognition algorithm for electronic travel aid,” Intelligent Control and Information Processing (ICICIP), 2013 Fourth International Conference on. IEEE, 2013.

3. Gong J, Jiang Y, Xiong G, et al. 2010, “The recognition and tracking of traffic lights based on color segmentation and CAMSHIFT for intelligent vehicles.” In IV, 2010, 23(3):431 - 435. 4. Wang W , Sun S , Jiang M , et al. 2017, “Traffic lights detection and recognition based on

multi-feature fusion.” Multimedia Tools and Applications, 2017, 76(13):14829-14846.

5. Charette R D, Nashashibi F. 2007. “Traffic light recognition using image processing compared to learning processes,” IEEE/RSJ International Conference on Intelligent Robots & Systems. 2007. DOI: 10.1109/ICMA.2007.4303519.

6. Cheng H D, Jiang X H, Sun Y, et al. 2001, “Color image segmentation: advances and prospects.”

Pattern Recognition, 2001, 34(12):2259-2281.

7. Nathan J, Henry G H, Cole B L. 1964, “Recognition of colored road traffic light signals by normal and color-vision-defective observers.” Journal of the Optical Society of America, 1964, 54(8):1041.

8. Wang C, Jin T, Yang M, et al. 2011, “Robust and Real-Time Traffic Lights Recognition in Complex Urban Environments.” International Journal of Computational Intelligence Systems, 2011, 4(6):1383-1390.

9. Varan S, Singh S, Kunte R S, et al. 2007. “A Road Traffic Signal Recognition System Based on Template Matching Employing Tree Classifier” Proc of the International Conference on Computational Intelligence & Multimedia Appl. 2007.

11. Behrendt K, Novak L, Botros R. 2017. “A deep learning approach to traffic lights: Detection, tracking, and classification” IEEE International Conference on Robotics & Automation. 2017:1370-7. DOI: 10.1109/ICRA.2017.7989163

12. Weber M , Wolf P , J. Marius Zöllner. DeepTLR: 2016. “A single deep convolutional network for detection and classification of traffic lights” Intelligent Vehicles Symposium. IEEE, 2016. 13. John V, Yoneda K, Qi B, et al. 2014, “Traffic light recognition in varying illumination using deep

learning and saliency map” IEEE International Conference on Intelligent Transportation Systems. 2014:2286-2291.

14. Liang X, Du X, Wang G, et al. 2018, “A Deep Reinforcement Learning Network for Traffic Light Cycle Control.” IEEE Xplore: IEEE Transactions on Vehicular Technology. 2018,68(2):1243-1253.