A Real-Time Driver Fatigue Detection System Based
on Eye Tracking and Dynamic Template Matching
Wen-Bing Horng* and Chih-Yuan Chen
Department of Computer Science and Information Engineering, Tamkang University, Tamsui, Taiwan 251, R.O.C.
Abstract
This paper presents a vision-based real-time driver fatigue detection system for driving safety. Based on skin colors, the driver’s face is located from a color video captured in a car. Then, edge detection is employed to locate the regions of the driver’s eyes, which are used as the templates for eye tracking in subsequent frames. Finally, the tracked eyes’ images are used for fatigue detection in order to generate warning alarms for driving safety. The proposed system was tested on a Pentium M 1.4G notebook with 512 MB RAM. The experimental results seemed quite encouraging and promising. The system could reach more than 30 frames per second for eye tracking, and the average correct rate for eye location and tracking could achieve 96.0% on five test videos. Though the average precision rate of fatigue detection was 89.3%, the correct detection rate could achieve 100% on the test videos. Key Words:Intelligent Transportation System, Driver Fatigue Detection, Eye Detection, Face
De-tection, Template Matching
1. Introduction
As the standard of living has improved in the past few decades, many families possess their own transpor-tation vehicles. Therefore, people often drive their cars for business or trip. However, long distance driving usu-ally makes drivers exhausted. Thus, thedriver fatigue problem has been an important factor of traffic accidents. In order to decrease traffic accidents, many countries have begun to pay attention to the driving safety problem. Some psychologists [1,2] have investigated the drivers’ mental states relating to driving safety. Furthermore, many other researchers [3-6,10-19] have proposed vari-ous auxiliary mechanisms to improve driving safety.
Cho et al. [3] designed a special car seat to improve the alertness of the driver by adjusting the temperature and humidity of the driver’s thighs and hips to reduce the degree of drowsiness. Chieh et al. [4] monitored the grip force change of the driver on the steering wheel to detect fatigue. Wilson and Bracewell [5] analyzed EEG
(Elec-troencephalogram) recordings by neural networks to mo-nitor alertness. Similar research was also performed by Vuckovic et al. [6] to detect drowsiness. However, these methods need some special sensors adhering to the hu-man body, which seems impractical during driving.
Other researchers based on image processing [7] and computer vision [8,9] techniques to detect driver’s fa-tigue from images captured by a camera to improve driv-ing safety. Some of these researchers [10-14] utilized in-frared CCD cameras to locate the driver’s face and eyes positions easily. However, infrared cameras are very ex-pensive, which makes them of limited usage. In addition, the influence of infrared light on human eyes is not known yet. To be practical, some other researchers [15-19] em-ployed ordinary CCD cameras to perform fatigue detec-tion. However, most of those driver fatigue detection met-hods suffer from the illumination change problem. In ad-dition, they might not be suitable for real-time applica-tions due to their complicated computaapplica-tions.
In this paper, we propose a vision-based real-time driver fatigue detection system based on ordinary CCD cameras to cope with the above drawbacks for practical *Corresponding author. E-mail: [email protected]
usage to improve driving safety. For the illumination change problem, we adopt the HSI color model to repre-sent face skin, not only because the model detaches the intensity from the hue of a color, but also because face colors have fixed distribution range on the hue compo-nent. Since eyes have the nature of sophisticated edges, we use the Sobel edge operator and the projection tech-nique to locate eyes’ positions. The obtained eyes’ im-ages are then used as the templates for eye tracking in subsequent frames, instead of using the static eye images in the database as used in [16], to enhance the reality and reliability of the templates to increase accuracy of eye tracking. Finally, the tracked eyes’ images are used to de-termine whether the eyes are open or closed, which is the basis of driver fatigue detection used in this paper.
The rest of the paper is organized as follows. In Sec-tion 2, we briefly review the related driver fatigue detec-tion methods based on ordinary CCD cameras. In Sec-tion 3, we present a new vision-based real-time driver fa-tigue detection system, including face detection, eye de-tection, eye tracking, and fatigue detection. The experi-mental results of the system are analyzed in Section 4. Finally, the paper is concluded in the last section.
2. Related Driver Fatigue Detection Methods
Based on Ordinary CCDs
In this section, we briefly review the related vision-based driver fatigue detection methods using ordinary CCD cameras. In 1997, Eriksson and Papanikolopoulos [15] proposed a method to detect driver fatigue from or-dinary videos. In this method, the symmetric property of the face is used to detect the facial area of the driver on an image. Then, pixel difference is performed to find the edges on the facial region, and horizontal projection is used to locate the vertical position of the eyes. Finally, a concentric circle template is used to locate and track the exact eyes’ positions. Although face symmetry is an ob-vious feature for an upright face, it usually fails to locate the correct face position when the face tilts, rotates, or is shadowed.
Instead of using symmetric central line method, Singh and Papanikolopoulos (1999) [16] later proposed an-other method to monitor driver fatigue. In this method, the Gaussian distribution of skin colors in the RGB color model is used to distinguish skin and non-skin pixels for
face detection. Then, the Sobel vertical edge operator is used for eye detection on the driver’s face. Besides, a set of eye images as templates in a database is used for eye detection and tracking. Although the Gaussian distribu-tion of skin colors based on the RGB color model is used to predict skin quite well, the method cannot get rid of the factor of illumination changes. In addition, the eye images in the database as templates may be quite differ-ent from drivers’ eyes, which will reduce the accuracy for locating eye positions.
In 2003, Wang et al. [17] proposed a method for mo-nitoring driver fatigue behavior. Like [16], the Gaussian distribution of skin colors in the RGB color model is used to locate a face region. Then, the face is binarized with a fixed threshold to locate the eyes’ positions with some geometric restrictions and to track the eyes by us-ing the Hausdorff distance template matchus-ing. Finally, the features of the eyes are analyzed by the Gabor filters, and the states of driver’s eyes are classified by neural networks. Since the eyes are detected with a fixed thres-hold, this system might not resist illumination changes.
In 2004, Wang et al. [18] proposed another method for driver fatigue detection by monitoring mouth move-ment. Similarly, the Gaussian distribution of skin colors is used to locate a face region. Then, a Fisher linear clas-sifier is used to detect the mouth of the driver, and a Kalman filter is used to track the mouth movement. How-ever, the mouth movement may not be a reliable feature for detecting drowsiness because the mouth shapes will also change when talking, eating, laughing, and so on. It is not easy to classify the mouth movement into a correct drowsy state.
In 2005, Dong and Wu [19] also presented a driver fatigue detection algorithm based on the distance of eye-lids. Instead of using the RGB color model, this method utilizes the Gaussian distribution of skin colors based on the YCbCr color model to detect the face region. Then, horizontal projection is used to locate eyes’ positions, and the gray scale correlation is used for eye tracking. Fi-nally, the distance of eyelids is calculated to detect fa-tigue. Although the method can resist the influence of il-lumination changes, it is not easy to distinguish the dis-tance of eyelids of closed eyes or open eyes by small dif-ference of eyelids’ distance, especially when persons have long eyelash.
fa-tigue detection methods using ordinary CCD cameras cannot resolve the illumination change problem. Besi-des, they might not be practical for real-time applications due to their complicated and intensive computations.
3. The Proposed Driver Fatigue Detection
System
The proposed system uses an ordinary color CCD camera mounted on the dash board of a car to capture the images of the driver for driver fatigue detection. The flow chart of the proposed fatigue detection system is depic-ted in Figure 1. The first image is used for initial face lo-cation and eye detection. If any one of these detection procedures fails, then go to the next frame and restart the above detection processes. Otherwise, the current eye images, as the dynamic templates, are used for eye track-ing on subsequent images. If eye tracktrack-ing fails, the pro-cesses of face location and eye detection restart on the present image. These procedures continue until there are no more frames. The detailed steps are described in the following subsections.
3.1 Face Detection
Digital images usually adopt the RGB color space to represent colors. However, any color in the RGB space not only displays its hue but also contains its brightness. For two colors with the same hue but different intensi-ties, they would be viewed as two different colors by the human visual system. In order to accurately distinguish skin and non-skin pixels so that they will not be affected by shadows or light changes, the brightness factor must be excluded from colors. Since in the HSI color model, hue is independent of brightness, this model is well su-ited for distinguishing skin and non-skin colors no mat-ter whether the face is shadowed or not. Thus, in this pa-per it is used for face detection.
The HSI color model has three components: hue, sa-turation, and intensity (or brightness). The detailed met-hod for converting a color between the RGB space and the HSI model refers to [7]. The following Equations (1) to (3) show the results of converting a color from the RGB model to the HSI model. For any color with three elements,R,G, andB, normalized in the range between 0 and 1, the formulas for converting to its corresponding H,S, andIcomponents are given as follows.
(1)
(2)
(3) When (B/I) > (G/I),H= 360°–H.
From the face images of the test videos, it is obser-ved that theHvalues of skin colors fall in the range be-tween 10°and 45°. Using this condition, it is easy to sep-arate the face region, as shown in Figure 2(b). By per-forming the vertical projection on skin pixels, the right
Figure 1. Flow chart of the proposed driver fatigue detection system.
and left boundaries can be found when the projecting values are greater than some threshold. In order to elimi-nate the region of the ears on the both sides of the face, the value of threshold is defined as 50 pixels through ex-periments. Similarly, by performing the horizontal pro-jection on skin pixels within the left and right bound-aries, the top and bottom boundaries can also be found, as shown in Figure 2(c). According to the normal posi-tion of the eyes in a face, it is reasonable to assume that the possible location of eyes will be in the block of the upper two fifths of the face region, as shown in Figure 2(d). Figures 3(a) and 3(c) illustrate that even under the case of uneven illumination on the faces during driving, the proposed method based on the HSI color model can still correctly segment the face regions.
3.2 Eye Detection
After locating the eye region, the original color eye region is converted into gray scale by Equation (4) as fol-lows (refer to [7]).
Y= 0.299R+ 0.587G+ 0.114B (4)
whereYis the gray value of a color (R,G,B). Then, the Sobel edge operator is used to compute the edge magni-tude in the eye region to find the vertical position of the
eyes. In order to save computation time, an approxi-mate calculation of the edge magnitude,G(x,y), is com-puted in Equation (5) as follows.
G x y( , )»Sx +Sy (5)
whereSxandSyare the horizontal and vertical gradient
values obtained, respectively, from the Sobel horizontal and vertical edge operators, and |Sx| represents the
abso-lute value ofSx. For detailed calculation, refer to [7].
After computing the edge magnitude of each pixel in the eye region, an edge map,E(x,y), is obtained by defin-ing a pixel (x,y) being an edge pixel if its edge magni-tudeG(x,y) is greater than some threshold,T, as defined in Equation (6).
(6)
whereblackand whitestand for the black pixel value and the white pixel value, respectively. The thresholdT is set to 150 via several tests. The left part of Figure 4(b) illustrates the edge map of Figure 4(a).
Performing horizontal projection on the edge map, the exact vertical position of the eyes can be located at the peak, as shown in the right part of Figure 4(b).
Some-Figure 2. Result of face detection.
times, it may need to smooth horizontal projections for locating peaks. After locating the vertical position of the eyes, as shown in Figure 4(b), the left and right eye posi-tions can be located by finding the largest connected components from the center of the eye region along the horizontal line of the vertical position of the eyes. In or-der to avoid marking noise, the total number of pixels in a connected component must exceed some threshold so that it can be regarded as a reasonable eye. After finding the connected component of an eye, a bounding box for the connected component on the original image is used to enclose the eye image, which is used as the template for eye tracking in the subsequent frames. Figure 5 illus-trates the bounding boxes of detected eye positions. 3.3 Eye Tracking
After finding the eye templates, they are used for dri-ver’s eye tracking by template matching. The searching area for matching is the original bounding box of an eye by expanding some reasonable number,t, of pixels, sayt = 10, in each of four directions. Consider an eye template g(x,y) locating in the position (a,b) of the face imagef(x, y). Equation (7) is used for the eye template matching.
(7) wherewandhare the width and height of the eye tem-plateg(x,y), andpandqare offsets of thex-axis and y-axis in which (a–t)p(a+t) and (b–t)q(b +t). IfM(p*,q*) is the minimum value within the sear-ch area, the point (p*,q*) is defined as the most match-ing position of the eye. Thus, (a,b) is update to be the new position (p*,q*).
The newly updated eye templates must satisfy the fol-lowing two constrains to confirm successful eye tracking:
(i) The ratio of width to height of each new eye tem-plate is between 1 and 4.
(ii) Since the movement of the driver’s head will not very big between each pair of consecutire frames (1/30 second), the distance between the newly ob-tained templates and the previous templates is less than 10 pixels.
Otherwise, the eye tracking will be regarded as failure and the system will restart the face and eye location pro-cedures.
3.4 Fatigue Detection
At this stage, the colors of the eyeballs in the eye templates are used directly for fatigue detection. Since the property that the eyeball colors are much darker is a quite stable feature, the eye templates are inverted (ne-gated) and then converted to the HSI color model. The original darker eyeballs become brighter ones in the in-verted image. According to the observation, the satura-tion values of eyeball pixels normally fall in the range between 0.00 and 0.14. This observation is used to dis-tinguish whether a pixel in an eye template is viewed as an eyeball pixel, as shown in Figure 6. When the eyes are open, there are some eyeball pixels, as shown in Figures 6(a) and 6(b). When the eyes are closed, there are no eye-ball pixels, as shown in Figures 6(c) and 6(d). By check-ing the eyeball pixels, it is easy to detect whether the eyes are open or closed. According to the experimental results as shown in Figure 7, the duration of a normal eye blink (the small peak) is usually less than 4 frames at the cap-ture rate of 30 frames per second. If the duration for an eye-close is more than 5 frames, the distance of the car
Figure 4. Results of edge detection and horizontal projection.
will move more than 5.5 meters at the speed of 100 kilo-meters per hour in a high way. It will be very dangerous for the driver to ignore the status of the road when driving under such a condition. For this reason, in our system it is defined that when the eyes close over 5 consecutive frames, then the driver is regarded as dozing off. Of course, this pa-rameter can be adjusted by users according to their needs.
4. Experimental Results
The proposed driver fatigue detection system used a Sony PC115 color DV camera to capture driver’s im-ages. The system was tested under the environment of Pentium M 1.4G notebook with 512 MB RAM. The for-mat of input video is 320´240 true color. After starting the system, it took less than 40 milliseconds for initial face location and eye detection. Once the eye templates were found, eye tracking could achieve more than 30 fra-mes per second. Recall that in this system, if the driver closes his/her eyes over 5 consecutive frames, then the driver is regarded as dozing off.
Table 1 lists the results of eye tracking from five test videos, as shown in Figure 8. The first four videos were
captured under different illumination conditions with dif-ferent people and backgrounds. The last video, video 5, was captured when driving around a parking lot at night-fall with large illumination change. The fieldTotal Fra-mesmeans the total number of frames in each video. Tracking Failure is the count of eye tracking failure. Correct Rateof eye tracking is defined as in Equation (8), which is the ratio of (Total Frames–Tracking Fail-ure) toTotal Frames.
(8) From Table 1, we can see that the correct rate of eye tracking is higher than 98.3% in the first four videos, and it still can reach 80.0% in the very strict environment of video 5. The average correct rate achieves 96.0% in the proposed system on these test videos.
Table 2 shows the result of driver fatigue detection on the five test videos. The fieldClose Eyesrepresents the number eye closing and then opening of the driver in the video.Real Dozingis the number of dozing off deter-mined by humans.Generate Warningis the number of fatigue detection that generates a warning alarm.False
Figure 6. Eyeball detection of open eye and closed eye.
Figure 7. Analysis of eye blinks.
Table 1.Result of eye tracking
Video 1 Video 2 Video 3 Video 4 Video 5 Total
Total Frames 2634 1524 2717 433 1399 8707
Tracking Failure 8 6 46 7 280 347
Positiveis the number of wrongly generated warning al-arms.False Negativeis the number of not detected fa-tigue.Correct Warningis the number of correctly detec-ted fatigue, which is equal to (Generate Warning–False Positive–False Negative).Correct Rate of fatigue de-tection is defined as in Equation (9).
(9)
Precision Ratefor fatigue detection is defined as in Equation (10).
(10) As can be seen from Table 2, the system could cor-rectly detect all fatigue states in all the test videos. In video 3, three mistakes made by the system are due to the interference of the driver’s hand, as shown in Figure 8(c). Even for the uneven illumination on the face in a moving car, as shown in Figures 8(e) to 8(h), the
pro-posed system could still correctly detect and track the driver’s eyes. In spite of this kind of interference and il-lumination changes, the average precision rate for fa-tigue detection for all videos could still achieve 89.3% in the proposed system.
5. Conclusion
This paper has presented a vision-based real-time driver fatigue detection system for driving safety. The system uses an ordinary CCD camera to capture the dri-ver’s images, which makes the method more practical. Tested on a notebook of Pentium M 1.4G CPU with 512 MB RAM, the system could reach 96.0% of average cor-rect rate of eye detection on five videos of various illumi-nation conditions, and could achieve 100% of correct rate of fatigue detection with precision rate 89.3%. The speed of this system could reach more than 30 frames per second for eye tracking, with each color frame of size 320´240. In summary, the proposed driver fatigue
de-Figure 8. Snapshots during tracking with different people under different illumination.
Table 2.Result of fatigue detection
Video 1 Video 2 Video 3 Video 4 Video 5 Total
Close Eyes 22 18 43 6 3 92 Real Dozing 03 04 15 2 1 25 Generate Warning 03 04 18 2 1 28 False Positive 00 00 03 0 0 03 False Negative 00 00 00 0 0 00 Correct Warning 03 04 15 2 1 25 Correct Rate 100% 100% 100% 100% 100% 100% Precision Rate 100% 100% 83.3% 100% 100% 89.3%
tection system could cope with the illumination change problem and could be suitable for real-time applications to improve driving safety.
Acknowledgement
The authors would like to thank the anonymous re-viewers for their valuable comments. This research was partially supported by the National Science Council, Tai-wan, ROC., under contract no. NSC 92-2213-E-032-031.
References
[1] Shinar, D., Psychology on the Road, John Wiley & Sons, Danvers, MA, U.S.A. (1979).
[2] Recarte, M. A. and Nunes, L. M., “Effects of Verbal and Spatial-Imagery Tasks on Eye Fixations While Driving,” Journal of Experimental Psychology: Ap-plied,Vol. 6, pp. 31-43 (2000).
[3] Cho, K. J., Roy, B., Mascaro, S. and Asada, H. H., “A Vast DOF Robotic Car Seat Using SMA Actuators with a Matrix Drive System,” Proc. IEEE Robotics and Automation,New Orleans, LA, U.S.A., Vol. 4, pp. 3647-3652 (2004).
[4] Chieh, T. C., Mustafa, M. M., Hussain, A., Zahedi, E. and Majlis, B. Y., “Driver Fatigue Detection Using Steering Grip Force,”Proc. IEEE Student Conference on Research and Development, Putrajaya, Malaysia, pp. 45-48 (2003).
[5] Wilson, B. J. and Bracewell, T. D., “Alertness Monitor Using Neural Networks for EEG Analysis,”Proc. IEEE Signal Processing Society Workshop on Neural Net-works for Signal Processing, Sydney, Australia, Vol. 2, pp. 814-820 (2000).
[6] Vuckovic, A., Popovic, D. and Radivojevic, V., “Arti-ficial Neural Network for Detecting Drowsiness from EEG Recordings,”Proc. IEEE Seminar on Neural Net-work Applications in Electrical Engineering, Belgra-de, Yugoslavia, pp. 155-158 (2002).
[7] Gonzalez, R. C. and Woods, R. E.,Digital Image Pro-cessing,Second Edition, Prentice Hall, Upper Saddle River, NJ, U.S.A. (2002).
[8] Davies, E. R.,Machine Vision: Theory, Algorithms, Practicalities, Third Edition, Elsevier, Oxford, UK (2005).
[9] Forsyth, D. A. and Ponce, J.,Computer Vision: A
Mo-dern Approach, Prentice Hall, Upper Saddle River, NJ, U.S.A. (2003).
[10] Gu, H., Ji, Q. and Zhu, Z., “Active Facial Tracking for Fatigue Detection,”Proc. IEEE Workshop on Applica-tions of Computer Vision,Orlando, FL., U.S.A., pp. 137-142 (2002).
[11] Ji, Q., Zhu, Z. and Lan, P., “Real-Time Nonintrusive Monitoring and Prediction of Driver Fatigue,”IEEE Transactions on Vehicular Technology, Vol. 53, pp. 1052-1068 (2004).
[12] Zhu, Z. and Ji, Q., “Real-Time and Non-intrusive Dri-ver Fatigue Monitoring,”Proc. IEEE Intelligent Tran-sportation Systems, Washington, D.C., U.S.A., pp. 657-662 (2004).
[13] Gu, H. and Ji, Q., “An Automated Face Reader for Fa-tigue Detection,”Proc. IEEE Automatic Face and Ge-sture Recognition, Seoul, Korea, pp. 111-116 (2004). [14] Ito, T., Mita, S., Kozuka, K., Nakano, T. and
Yama-moto, S., “Driver Blink Measurement by the Motion Picture Processing and its Application to Drowsiness Detection,” Proc. IEEE Intelligent Transportation Systems,Singapore, pp. 168-173 (2002).
[15] Eriksson, M. and Papanikolopoulos, N. P., “Eye-Tracking for Detection of Driver Fatigue,”Proc. IEEE Intelligent Transportation Systems,Boston, MA, U.S.A., pp. 314-319 (1997).
[16] Singh, S. and Papanikolopoulos, N. P., “Monitoring Driver Fatigue Using Facial Analysis Techniques,” Proc. IEEE Intelligent Transportation Systems, To-kyo, Japan, pp. 314-318 (1999).
[17] Wang, R., Guo, K., Shi, S. and Chu, J., “A Monitoring Method of Driver Fatigue Behavior Based on Machine Vision,”Proc. IEEE Intelligent Vehicles Symposium, Columbus, Ohio, U.S.A., pp. 110-113 (2003). [18] Wang, R., Guo, L., Tong, B., and Jin, L., “Monitoring
Mouth Movement for Driver Fatigue or Distraction with One Camera,”Proc. IEEE Intelligent Transpor-tation Systems,Washington, D.C., U.S.A., pp. 314 -319 (2004).
[19] Dong, W. and Wu, X., “Driver Fatigue Detection Ba-sed on the Distance of Eyelid,”Proc. IEEE VLSI De-sign and Video Technology,Suzhou, China, pp. 365 -368 (2005).
Manuscript Received: Dec. 29, 2006 Accepted: Jul. 25, 2007
