In this chapter, a new method named MERCY for eye movement classification was intro- duced. MERCY can be seen as an extension of the Bayesian Mixture Model approach proposed by Tafaj et al. in [62] which was already applied in dynamic driving scenar- ios. However, for implementations on common RCP and HiL tools in the vehicle and for applications in conditionally automated driving scenarios, the Bayesian Mixture Model is too sophisticated and lacks the adaptability to the individual gaze behavior. In contrast, it was shown that MERCY exceeds state-of-the-art approaches including the Bayesian Mix- ture Model for eye movement classification in both classification performance and general adaptability based on half a million randomly generated data samples and a thorough con- ditionally automated driving study. Despite excellent classification performance, MERCY
is based on simple mathematics and therefore is easy to implement. For demonstration purposes and to verify MERCY online in the vehicle, it was implemented in Simulink including a graphical user interface and transferred to a testing vehicle. Due to the high adaptability of MERCY, the task-individual difference was shown to be significant be- tween the viewing behavior of subjects performing secondary tasks and idle subjects, both driving in a conditionally automated setting. The findings suggest that the eye movement behavior during changing tasks varies constantly and therefore the threshold for the clas- sification between saccades and fixations is varies, too. This indicates the necessity for adaptive thresholds for this task, which until now was an unresolved topic in the field of eye movement classification.
Since MERCY uses sample mean and variance estimators, a reliable estimation of the variance first requires a good estimation of the sample mean. That means that in case of sudden changes in eye movement behavior, the variance is estimated insufficiently as long as the mean has not approximated the actual mean, causing an overshoot of the intersection parameter. The error of the estimation of the sample mean impacts the estimation of the variance quadratically. A possible solution could be a correction function depending on the gradient of the sample mean. Moreover, MERCY is updates only the parameter set Θf or
Θsof the estimated GMM of the current classification result. In case of a large overlap of
the two Gaussian distributions, e.g. due to poor initialization values, the incorrect param- eters are often updated. Since the total error of the falsely classified data samples can be estimated, this error should be considered in the estimation of the parameters of the model in terms of error minimization. In this way, both parameter sets Θf and Θscan be updated
in every iteration.
In the following chapters, MERCY will be applied as the subsystem for some of the pro- posed algorithms for Eyes-on-Road detection and driver-activity recognition.
While driving in an automated setting, drivers have the opportunity to take their eyes off the road, e.g. to perform secondary tasks, since there is no need for a detailed monitoring of the traffic environment. To determine whether the driver is focusing on the road, various systems were introduced in the literature and are even available in series vehicles. These systems and algorithms are subsumed by the Eyes-on-Road concept. When performing secondary tasks in conditionally automated driving scenarios, many drivers tend to gaze towards the street, the instrument cluster, or the vehicle’s mirrors. These gazes allow a reorientation of the driver in terms of the current traffic situation and, therefore, have an impact on the take-over quality in take-over situations. Especially the detection of these typically short Eyes-on-Road gazes is challenging in real-world traffic environments due to various lighting conditions or not visible eyes due to large head angles. These challenges cannot be solved solely by improved hardware and computer vision algorithms at the mo- ment. Hence, in this chapter novel algorithms based on given eye- and/or head-tracking signals will be introduced to improve the Eyes-on-Road detection. After a phenomenolog- ical description of Eyes-on-Road gazes and a summary of existing methods for Eyes-on- Road Detection in the first two Sections 4.1 and4.2 of this chapter, it will be shown in Section 4.3that a relative gaze direction can enable a highly accurate Eyes-on-Road de- tection without any kind of calibration. For the case of missing eye gaze signals, e.g. due to large head angles or camera systems without eye-tracking, an architecture for detecting Eyes-on-Road gazes solely on the head movements is introduced in Section 4.4. As in the previous chapter, all algorithms will be analyzed with regard to their applicability in a real-world testing vehicle with a close-to-production camera system at the end of this chapter.
4.1 Visual Attention and Eyes-on-Road Gazes
The majority of sensory perception in daily life, in detail about 80%, is received over the visual sensory channel, which corresponds to a transmission rate of about 6.5 MB/s [71]. It is hardly surprising that according to Sivak this statement can be transferred to driving situations [72]. Hence, visual distraction of the driver is considered to be one of the most critical conditions and frequent reasons for near- and actual crashes in the traffic environment [73]. Multiple studies were conducted to examine the correlation between visual distraction and driving performance, e.g. Wierwille and Tijerina in [74] or Jamson and Merat in [75]. Common reasons for visual distraction in non-automated vehicles are secondary tasks or stimuli from the environment. For example, Greenberg et al. [76] performed a thorough investigation of the impact of various versions of using a mobile
phone while driving, such as hands-free and hand-held phone dialing. The study showed that the rate of missed events in front of the vehicle is similar for hands-free and hand-held devices due to the increased visual demand of the secondary tasks. In another study, Wallace [77] showed that stimuli from outside of the vehicle can be a dangerous threat to the safety of the driver and other traffic participants. Especially advertisements and signs at junctions or on long monotonous roads may distract the driver significantly.
At this point the question arises, how visual distraction or attention can be assessed in the traffic environment. By fixating any location with the fovea centralis of the eyes, drivers may focus their visual attention to the chosen target. However, it should be noted that the eyes are not able to process all the information inside the visual field of view at once. In- stead, different properties of the same target, such as the color or shape, can be observed sequentially [78]. By performing eye movements, such as saccades, the visual focus can be shifted to other targets. Hence, the gaze behavior, in detail the location and the dura- tion of a fixation as well as the performed movements of the fovea, can be associated with visual attention or visual distraction, respectively. Many studies have already proven this close relation between gaze behavior and visual attention, e.g. refer to Frischen [79] for an overview of this topic in daily life social interactions or Chapman and Underwood [80] with regard to the traffic environment. Besides the fovea centralis, studies show that the driver’s peripheral vision may also provide essential information concerning visual atten- tion. For example, Summala et al. [81] showed that experienced drivers are able to keep the lane just by depending on their peripheral vision. Further, on an empty highway in the dark, peripheral vision is sufficient to perceive differences in light intensity indicating ap- proaching traffic. However, since this study focuses on drives in daylight and it is assumed that none of the test subjects has experience applying peripheral vision in conditionally au- tomated driving scenarios, the influence of peripheral vision on the driver’s visual attention will be neglected.
Note that the driver’s gaze behavior and visual attention are subject to individual and situational differences. For example, Konstantopoulos et al [82] compared the visual search strategies of drivers’ with different levels of driving experience and showed the modification of the gaze behavior of the drivers according to visibility conditions. These results show that both external factors such as visibility conditions and internal factors such as driving experience influence the gaze behavior and, therefore, the visual attention of the driver and need to be taken into account. Typical gaze parameters used for detecting visual inattention in the driving environment are the number or frequency of off-road gazes and the mean and maximum duration of the off-road gazes [83], [84], [85]. An important step for clarifying the use of the off-road gazes was introduced by Peng et al. in [83]. An initial separation of drivers into two classes based on gaze behavior, namely low-risk and high-risk, was possible based on the maximum duration of their off-road gazes.
In conditionally automated driving scenarios, the driver is no longer responsible for the driving task and, therefore, may look away from the road the entire time. Nevertheless, the gaze behavior still contains crucial information about the visual attention of the driver, since interesting phenomena may occur when the driver performs visual secondary tasks.
In such situations, most drivers tend to gaze towards the road and the mirrors for a few seconds before continuing with the interrupted task as illustrated in Figure4.1. These will be subsequently be referred to as Eyes-on-Road gazes or just EoR gazes. The reasons for these EoR gazes are not sufficiently investigated. One possible explanation for this behavior could be other traffic participants attracting the attention of the driver or fading attention towards the secondary task. However, it can also be assumed that drivers who are used to driving manually perform gazes towards the driving task to reorientate themselves. This assumption is further underpinned by the results of Zeeb et al. in [11]. In a large- scale driving simulator study of 107 participants, the drivers experienced multiple take- over situations while driving in a conditionally automated scenario. Zeeb et al. categorized these drivers into three groups similarly to [83] and analyzed among other things the impact of the EoR gazes on take-over performance. It could be shown that drivers who tend to perform many and longer EoR gazes are able to take-over in an appropriate way and usually in a shorter time interval than drivers who are focused solely on their secondary task. Based on these findings, parameters of the EoR gazes will be used as input for the later classification of the take-over readiness.
(a) Eyes off-road
(b) EoR gaze
Figure 4.1:The driver performs an EoR gaze by interrupting the secondary task and looking up at the road ahead before continuing with the task. In both images, two black-white markers mounted beside the steering wheel can be seen which may be used for detecting gazes in the corresponding areas.
In summary, the driver’s visual perception provides the most crucial information for driv- ing scenarios, presaging the severe consequences of visual distraction. The visual attention or distraction can be assessed over the driver’s eye movement behavior and gaze direction which are subject to inter- and intra-individual differences. These findings apply for both non-automated and conditionally automated driving scenarios. As previous studies show, EoR gazes seem to be a promising indicator for visual attention, situation awareness, and maybe also for the driver’s take-over readiness in conditionally automated driving scenar- ios.