International Journal of Advanced Robotic Systems
Human Adaptive Mechatronics
and Human-System Modelling
Regular Paper
Satoshi Suzuki
1,*, Hiroshi Igarashi
2, Harumi Kobayashi
3, Tetsuya Yasuda
3and Fumio Harashima
41 Department of Robotics and Mechatronics, Tokyo Denki University, Japan 2 Department of Electrical and Electronic Engineering, Tokyo Denki University, Japan 3 Division of Information System Design, Tokyo Denki University, Japan
4 Tokyo Metropolitan University, Japan
* Corresponding author E-mail: [email protected]
Received 1 May 2012; Accepted 8 Jan 2013 DOI: 10.5772/55740
© 2013 Suzuki et al.; licensee InTech. This is an open access article distributed under the terms of the Creative
Commons Attribution License (http://creativecommons.org/licenses/by/3.0), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.
Abstract Several topics in projects for mechatronics studies, which are ʹHuman Adaptive Mechatronics (HAM)ʹ and ʹHuman‐System Modelling (HSM)ʹ, are presented in this paper. The main research theme of the HAM project is a design strategy for a new intelligent mechatronics system, which enhances operatorsʹ skills during machine operation. Skill analyses and control system design have been addressed. In the HSM project, human modelling based on hierarchical classification of skills was studied, including the following five types of skills: social, planning, cognitive, motion and sensory‐motor skills. This paper includes digests of these research topics and the outcomes concerning each type of skill. Relationships with other research activities, knowledge and information that will be helpful for readers who are trying to study assistive human‐mechatronics systems are also mentioned.
Keywords Mechatronics, Skill Analysis, Assistive Control
1. Introduction
There are a growing number of disciplines for human‐ friendly mechatronics design such as Human‐factors
engineering, Ergonomics design and Human centred design. These technologies have been utilized for the design of mechatronics systems and gadgets in our daily life. No matter how well mechatronics are designed, however, there are always situations where the user has to adapt to a mechanical system on one level or another. The reason is that usually the machine itself is not designed so as to adapt to each human user, although human users can adapt to each mechatronics system. Therefore, to resolve such a seemingly contradictory situation, the concept of Human Adaptive Mechatronics was launched in 2004 [1]. A HAM‐project was started as a 21st century Centre of Excellence (COE) programme supported by MEXT (Japanese Ministry of Education, Culture, Sports, Science and Technology) during 2004‐ 2008 [2,3]. Required functions for HAM are the ability to adapt to an operator’s skill and assist them. Therefore, human‐skill evaluation methods and assistance control methods (for instance, a tiny force assist control on an haptic system [4], an adaptive impedance control [5, 6], an assistant control system recovering against inadequate human operation [7] and safe manual control [8]) were studied and several applications, such as a surgery support robot system [9], were developed.
The conceptual meaning of ʺskillʺ has been, however, dealt with ambiguously. In other words, systematic approaches considering skill type have not been discussed sufficiently in the HAM project. Hence, as the next challenge after the HAM project, new research into `human‐system modellingʹ was carried out during 2008‐ 2010 for the systematization of skill evaluation and assistance control [10]. The main purposes of this are ʹthe establishment of systematic human system modellingʹ and ʹthe accumulation of assistance methods and skill evaluation methodsʹ. For the first purpose, a hierarchical classification of skills was proposed, as shown in Fig. 1. This classification was based on existing human models (such as a Normanʹs seven stage model, a model human processor and Rasmussenʹs skill/rule/knowledge model) and outcomes obtained from the HAM project [11].
Figure 1. Hierarchical classification of skills
The hierarchy consists of the following skills: social‐ (S1), planning‐ (S2), cognitive‐ (S3), motion‐ (S4) and sensory‐ motor (S5) skills. The S5, a sensory‐motor skill, is the lowest and relates mainly to voluntary motion. This skill concerns cooperation between the neural system and the musculo‐skeletal system inside the human body. S4 (motion skill) relates to the continuous execution of each body part motion. For example, the sequential execution of finger, hand and arm motions when using a hand tool. S3 is cognitive skill. This mainly concerns recognition of circumstance and includes a wide range of cognitive issues, such as the understanding of meanings. S2 (planning skill) relates to understanding a whole task process and recognising sub‐tasks within the whole task. This skill also relates to the task management of sub‐ tasks, which requires consideration of eventsʹ causality and time scheduling. S1 is social skill, which relates to communication, conversation, negotiation and the estimation of other intentions.
Of course this hierarchical classification is not perfect; however, such classification will be helpful for research concerning human skills and for the design of human assistive systems. For instance, when a job involves assistance system design, the system designer will be able to find a better approach of how to deal with each skill
type after clarifying the priority of these skill types. By referring to similar methods categorized as each skill type, concrete solutions for the design of an assistive system can be obtained. Therefore, this paper introduces research activities, knowledge and information, which will be useful for readers who are trying to study assistive mechatronics systems. In particular:
Digests of the authorsʹ research activities and outcomes concerning each skill type are shown and Items and factors that should be considered for each
skill are explained.
The organization of this paper is as follows. Section II introduces experimental tasks, which were designed or used in the authorsʹ study. These tasks are explained in advance in order to make the following topics easier to understand for readers. In Section III, analyses and outcomes relating to each skill type are explained with an introduction to the background and other existing studies. Section IV introduces a comprehensive analysis that addresses the social skills involving other levels. Finally, Section V is a conclusion of the research.
2. Experimental tasks to investigate individual skill levels Research on the human‐system modelling project has been carried out basically through the following process: 1) design of experimental tasks concerning the hierarchical classification of skills, 2) collection of data of human behaviour measured using these experimental tasks, 3) proposal of a skill evaluation method by analysis of the collected data, 4) suggestion of how to design an assistance system by utilizing the obtained knowledge and 5) verification of the presented assistance system by using the experimental task again. In order to help the reader understand the topics described in the latter sections, each experimental task is explained in this section.
A. Stick stabilization task (ss‐task)
Stick balancing is a good example of a method to study a voluntary motion mechanism, since the participant has to recognize the status of the stick (that is a controlled object) and control his/her own hand according to the status. Many studies on stick balancing or manual control of a pendulum have been reported [12‐19]. The following facts about a skilled stick balancer are known:
The distribution of changes in hand velocity is a truncated Levy distribution [15].
The power spectrum of fluctuations in the stickʹs height shows two scaling regions with two different power laws [14, 18, 20].
About 98% of corrective movements occur faster when compared to the time delay of human perception [16].
Since knowledge of the skill properties required for stick
balancing is well known, a stick balancing task was
adopted for the authorsʹ study and investigation of the
sensory‐motor skill. Since our primary research concern is
human assistance in a machine operation, an
experimental system to stabilize a virtual computer‐
graphics (CG) stick by manipulation of a machine was
developed, as shown in Fig. 2(a). The experimental
system consists of two units: a real‐time computer
graphics generation of a virtual pendulum and a haptic
interface device. The participant manipulated a grip fixed
to the slider of the interface device and the force exerted
by the participant was measured by a sensor embedded
in the slider. The motion of the virtual pendulum was
computed in real time using the measured force. A
process and principle for acquiring the sensory‐motor
skill were investigated by analysing the relationship
between the force and status of the virtual pendulum by
the system identification method along with a brain
monitoring method
Figure 2. Experimental setup of stick balancing task (a) and the
head probe of the NIRS system (b).
An optical topography system, which is a non‐invasive
measurement of changes in the concentration of oxy‐ and
deoxy‐haemoglobin using near‐infrared spectroscopy
(NIRS), was used as a brain monitoring method, as
shown in Fig. 2(b). Since “the hemodynamic response is
partly related to neuronal activity [21]” the activation
strength in each local brain area can be estimated by
detecting changes in the concentration. NIRS is robust
against electrical noise because it utilizes optical
measurement, which allows it to measure the brain
activation of natural behaviour in a non‐restrictive
environment, such as speaking [22], reading [23] and
language recognition [24]. Because of the admissibility to
participantʹs natural body motion and the robustness
against electrical noise from other machines manipulated
by him/her, NIRS was adopted for our study.
B. Remote radio control task (rr‐task)
In this task, a participant executes tele‐operation of radio‐
controlled constructing machines [25]. This task was
devised to analyse the hand and eye motion of the
operator in order to investigate cognitive and planning
skills. The purpose of the operation was basic soil
excavation work, as shown in Fig. 3(a).
Figure 3. Radio controlled construction equipment in a work
area (a) and the operation console (b).
The field included three drilling sites, one unloading site,
a motorable road and restricted areas. The operator
manipulated both an excavator and a truck and used
console switches to move the machines to the drilling site,
collect sample pieces with the excavator, load the pieces
on the truck bed and carry them to the unload site by the
truck. He/she was required to optimize the task
scheduling by considering shortening the task time.
Wireless cameras on the excavator and the truck captured
video images and displayed the images on monitors for
the operator, as shown in Fig. 3(b). To measure the
positions of the machines in the field, infrared LED
markers were placed on the work area and were attached
to the machines and were observed by a camera attached
to the ceiling. Their positions and directions were
computed by the Labview system with an image‐
processing module. The operation of switches on the
console was also recorded by the A/D and DIO interface
in the Labview system. Eye motion of the operator was
also measured and utilized for skill analysis.
C. Dynamic driving task (dd‐task)
This task is a kind of driving simulator that imitates the
dynamics of a hovercraft on a giant slalom course. A
hovercraft was adopted since its manipulation was difficult
for almost all participants in the beginning. Fig. 4 shows the
whole course. The course consists of side walls, a start‐line, a
goal line and 15 check‐flags. Participants were required to
propel the hovercraft to the goal passing through all 15 flags.
At that time, he/she was requested to consider the following
two points: whether to be as fast as possible and as close to
the flags as possible. The operator controls the hovercraftʹs
thruster forces by manipulating a joystick, as shown in Fig.
5(a). Logs of the joystick manipulation, hovercraft status and
gaze position were measured and the human control
characteristics in training were analysed.
Figure 5. An experimental scene of the dynamic driving task (a)
and a camera image captured by the gaze measurement system
(b).
D. Cooperative carrying task (cc‐task)
This task was designed to collect data of human
behaviour in order to analyse social and planning skills.
Conversation among participants and their eye motions
were recorded and brain activity, brain waves and the
robotsʹ manipulation data were measured during the
task. Fig. 6 shows the experimental scene and the
recorded image for gaze analysis.
Figure 6. An experimental scene of the cooperative carrying task
(a) and a sample of recorded image (b).
In the cooperation task, three participants worked in the
same virtual space. Each participant sat in front of each
monitor, used a joystick to manipulate their own virtual
mobile robot, cooperated with the other operators and
conveyed three boxes to target places. Fig. 7 shows an
overhead view of the virtual space. The virtual space
layout was intentionally designed so as to require
cooperation, such as simultaneous action to rotate the
box, to induce self‐motivated operation among
participants.
Figure 7. Overhead view of the virtual space for cooperation‐
carrying task
3. Analysis and its outcome
Results obtained by the experimental tasks, which were
mentioned in Section II, are explained in this section with
respect to each skill type.
A. Sensory‐motor Skill (S5)
Sensory‐motor control is an essential human
processing/motion system, which involves haptic and
visual perception, information processing in the brain
and an activation of muscles in the arm and leg. In the
manipulation of most mechatronics systems, visual
information is mainly utilized. Specifically for skilled
manipulation of a machine, adequate ability of a
visuomotor control is required for the operator to control
their own arm and hand in order to adjust motion of an
interface device such as a handle or a lever depending on
a change of scene. As studies of visuomotor control,
numerous studies such as a linear servo control model
[26], a PID‐based time‐variant model having randomness
[27], and an optimal control model [28] are known, and
they have been utilized for a design of human‐machine
systems. Although the focuses of those models differ
according to its objective, the main points of the
discussion about visuomotor control are (a) time‐delay
compensation, (b) learning and (c) estimation.
Concerning point (a), delays lie between about 30ms for a
spinal reflex and up to 200‐300ms for a visually guided
response [29]. Since a delay decreases the stability of the
whole system from a control engineering viewpoint, the
visuomotor control cannot work well without some
compensation for delay. The Smith‐predictor is often
used as a human model to compensate for time‐delays in
the nerve system [30] when a target system controlled is
stable. However, if the controlled system is unstable, the
Smith‐predictor is not adequate for the human control
model since an estimation error in the Smith‐predictor
does not decrease theoretically [31]. In that case, it is said
that a human model behaves like a forward model in a
sensory pre‐processing loop. In short, there is more than
one way to explain the human visuomotor process.
Regarding the discussion of point (b), a feedback‐error‐
learning model is widely accepted [32]. The concept of
the model is that a forward model‐based controller gives
the most suitable explanation for the visuomotor control
after learning [33]. The forward model is considered to be
a basic mechanism for estimating the next state of the
body motion [31]. This corresponds to point (c).
The mechanisms of the visuomotor control are explained
as mentioned above. However, few models can explain
the stages of the learning process clearly. Therefore, in
our study the changes in brain activities and control
characteristics of the human operators were investigated
from the beginner‐level till the expert‐level by using the
ss‐task. First, participants were trained in the ss‐task and
hand motion/force was measured. At the training stage,
at least ten trials a day (over more than five days) were
imposed for each participant. Written consent and ethical
approval were obtained before the examinations (all
experiments were done after consent and ethical
of changes in the hand velocity of participants, the skill levels of the participants were classified into high‐ performance (HP), moderate‐performance (MP) and low‐ performance (LP) by referring to Cabreraʹs studies [14]. Finally, using the data of the HP participants, the control characteristics of the skilled stick balancers were identified. In particular, if a human is assumed to be a controller that perceives an inclination angle of the pendulum
and outputs the hand force fh, the following control law in consideration of the time shift can be defined:f (t) kh (t ) k(t ) kd, (1) where kand kare constant gains to be estimated and t
is time. Here, and are time shifts andk is d the drift term. To search for the best values of the time shifts, the error index e, which was computed by averaging the identification error of |f (t) f |h h , was checked. Here,fh is the estimated force that was computed using the identified parameters k, kandk . d The best combination of andwas determined by changing
them to : 1 1 and : 1 1. Fig. 8 shows the strength distribution of the identified coefficients k, k,
d
k and e. The vertical and horizontal axes on each graph are and , respectively. The white (black) area in
each graph indicates the large (small) value.
Figure 8. Strength distribution of the identified coefficient: (a)
k, (b) k, (c) kd and (d) e.
As the drift term kd is sufficiently smaller than any of |k | or |k | , Eq. (1) appears to be appropriate as the
form of the control law of the expert. Moreover, the magnitude of |k| was larger than that of the other terms; hence, it transpired that the expert was paying attention to the change in velocity more than posture information. Furthermore, Fig. 8 shows that the error index in the vertical axis is smallest when 0.18 and that the error index in the horizontal axis is smallest when
0.04
. This negative sign of is interpreted as the
prediction concerning the measured velocity information. In conclusion, the expert recognized the posture of the controlled object about 0.18 seconds after observation and predicted the velocity about 0.04 seconds before by using oneʹs internal model. It can be surmised that the prediction based on an internal model of the controlled object is key to the skilled visuomotor control.
Next, the measured NIRS data were analysed to find the brain property of the skilled balancers. Generally, the primary motor cortex (MsI) and the primary somatosensory cortex (SmI) are important for voluntary motion and the movements of most muscles in the body are controlled by local regions in these cortices. Furthermore, the supplementary motor area (SMA) and the premotor cortex (PMC) relate to generating command of the voluntary motion from the generated intentions. Locations of these brain areas and the measurement position of the NIRS headset are illustrated in Fig. 9(a) and 9(b), respectively. The International 10‐20 measurement system, which is an application method that uses the electrodes of an electroencephalogram and is based on the distance between the nasion and the inion of the scalp, was used to determine the position of the probe. Changes in the concentration of total haemoglobin (sum of oxy‐ and deoxy‐heamoglobin) were measured using an ETG‐4000 system (Hitachi Medical Corporation, Tokyo, Japan). Reflections of lasers in the near infrared were measured 10 times during each sampling interval and the measured data was output every 100ms by averaging these reflections to attenuate noise effect.
Figure 9. Positional relation of side view of the brain (a) and the
corresponding position of NIRS headset (b).
The 48ch data of total haemoglobin were analysed using principal component analysis (PCA). As a result, a topography map in the brain showed the difference between the HP, MP and LP participants, as shown in Fig. 10. When we investigated activation of the HP participantʹs brain, activation of the right and left regions corresponding to the eyeball in the primary motor cortex (MsI) could be recognized (the corresponding area is labelled using (a3) on images (a) in Fig. 10). Further activation in the right arm and hand regions (that are located at MsI in the left hemisphere) was not strong. On the other hand, strong activations for the MP and LP participants were found in the narrow areas of the torso and hip, as shown by (b1) and (c1) in Fig. 10. From these facts, it would be appropriate to think that activation of
the ocular motor area is a characteristic of the HP participant. This means a skilled person concentrates on observing more than controlling.
Figure 10. Topographical images of stick balancers in the three
skill levels
According to a study by Cabrera, a skilled stick balancer performs the on‐off intermittency control (the drift‐and‐ act control). That is, an expert at stick balancing moves the hand faster than the time delay of human perception using a feed forward control and he/she changes this into feedback control when the pendulum link tilts largely. Existence of the on‐off intermittency control means that a human is not a simple continuous‐time controller but a complex of controllers that are switched depending on the circumstances. However, from the above‐mentioned identification results, it was shown that stick balancing control can be described well by one linear model involving two types of time shifts for the position and velocity, as defined by Eq. (1). This indicates that the on‐ off intermittency control can be approximated by one control law involving two types of time shifts. This formula will be effective as one for human controller models to design human‐assistive mechatronics [34]. B. Motion Skill (S4)
The general motion of a person consists of a sequence of simple motions including voluntary motion. Therefore,
the performance of the action decreases if transitions between motions are not sufficient. To investigate the skill of such general motion, characteristics of the transition and the motion control in operators were analysed using the dd‐task, which is more general than the ss‐task from the point of view of a general machine operation. Using gaze measurement during machine operation, all the operation sequences were segmented into piece‐wise linear intervals by checking the saccadic eye movement of the operator’s change in gaze position from one target to another.
An illustration that explains the spatial relation in the dd‐ task virtual space is shown in Fig. 11, where is a relative angle of the hovercraftʹs travelling direction to the closest flag (=first flag) and d is the relative distance between the hovercraft and the first flag. Here, is a change of the elevation angle against the fore object and is known as optic flow, which is an effective index for analysing driving behaviour [35]. Note that d and are geometrical variables in a virtual 3D space and the is a variable against the 2D monitor screen. This index is based on the concept that a human utilizes a change of elevation angle to estimate a change of perspective.
Figure 11. Conceptual illustration of spatial relation around a
virtual hovercraft of the dd‐task.
The control law of the operator was identified using the following piece‐wise linear model with three inputs (, , d) and one output (a rotation command to the hovercraft, ).
1 2 3
[t ] a [t ] a [t 1] a [t 2]
1 2 3
b [t ] b [t 1] b [t 2]
c d[t ] c d[t 1] c d[t 2]1 2 3 (2) where tʹ is the time when the operator saw the second flag first and a ,bi i and ci (i=1,2,3) are gain parameters to be identified. After identification, the transition of the ratios in these gain parameters, which were computed as follows, were investigated to check the learning process: : 100 ave(a / (a b c)) (3)
: 100 ave(b / (a b c)) (4) d: 100 ave(c / (a b c)) (5)
where a :
3i 1|a |/3i ,b :
i 13|b |/3i ,c :
3i 1|c |/3i and ave (*) means an average operation.
Five trials a day during five days were carried out by five novice participants and the obtained data were analysed by the aforementioned procedure. Table 1 summarizes the correlation factors between each ratio
and the manipulation performance (sum of minimum approach distances to flags, Ef) for each participant.
Table 1. Correlation coefficients of gain ratio against a manipulation performance
Since the table shows that the correlations of Ef and are all positive, the gain ratio of decreases as the participant becomes more skilled, because Ef decreases as the trial increases. In other words, a weakening of the utilization of the elevation angle information is related to an increase in skill. Similarly, other correlation factors of (Ef vs
) and (Ef vs
d) are all negative. Hence, the participants increased the gains to the relative angle and the distanced. These variables are not observable directly to the operatorʹs eye since and d are variables in the virtual 3D space. Therefore, it is interpreted that a skilled operator can recognize the virtual 3D spacial relation from the 2D image information displayed in the monitor. This fact is useful for evaluating skill in tele‐operations.
C. Cognitive Skill (S3)
For the operation of mechatronics, especially for vehicle machine operation, an ability to understand the environment around the operated machine, that is cognitive skill, is required. Although research regarding cognitive skill essentially belongs to the psychological field, how to utilize it for a mechatronics system should be considered for mechatronics engineers. Therefore, our study concerning cognitive skill was promoted to find a better methodology for a design of a mechatronic assistance system by utilizing the rr‐task.
First, we focused on the gaze from which the cognitive activity might be estimated from outside. The cognitive skill for machine operation was investigated through analysis of the eye motion. As a result, video analysis of the gaze pattern in the training process showed the following facts: at the early stage, the operator tends to watch more than one monitor on the console desk in order to watch the target object; at the skilled stage, he/she watches a space and the ground through one main monitor and does not see the target directly [36]. These
facts are interpreted as follows. The operator acquires proficiency in environmental cognition and does not pay attention to objects directly, then uses visual perception mainly to check positional relation inside the workspace. Such characteristic behaviour can be utilized to estimate cognitive skill level from eye motion.
Second, a machine function to evaluate an operatorʹs cognitive skill level qualitatively was studied. One powerful computational tool for this is ontology, which is a network model to describe a knowledge structure. Originally, ontology is a term used in philosophy and is the study of the nature of being, existence or reality in general. In more recent computer science, this concept is used as the basic category of being and relations that can be shared by both a human and a computer. Building up ontology consists of two steps: (1) extraction of relations of `concept classesʹ as a `semantic linkʹ and (2) repetition of segmentation of each concept class into small classes. In our research, structures of the knowledge required to perform the cc‐task were built by the Action Fast Method [37] and the methodology to apply it to a design of human‐machine systems was studied. For instance, a part of the cc‐task ontology is shown in Fig. 12. Since factors and causality that relate to present statuses can be estimated by referring the link network of the concept classes in the obtained ontology, the network structure of ontology can be utilized to guess an operatorʹs (future) intention and to evaluate their cognitive skill. An applied example is mentioned in Section 3.D2.
Figure 12. A part of the cc‐task ontology
D. Planning Skill (S2)
D1. Evaluation of skill from the reaching action
After mastering both voluntary motion control (S5) and the combination of their motions (S4), a planning skill to manage sub‐actions is required. Since planning is a reiteration of judgement and the execution of actions, superiority and inferiority in the connections of these two processes appears as a difference in the level of the planning skill. Conversely, the detection of unnatural motions may be effective in evaluating the level. In fact, such unnatural motion is known as microslip, which is a
jerky movement in a hand reaching action. By Reedʹs definition, there are four types of microslip: trajectory change, hesitation, change of the hand shape and touching objects [38]. It was reported that microslips appear more frequently under high cognitive loads and complex circumstances [39]. In our previous study, the relationship between task performance and microslips was investigated through a coffee‐making test and the strong correlation was confirmed statistically [40]. This result indicates that a human’s planning skill can be estimated by observing the hand motion.
Therefore, using the rr‐task, the planning skill of a machine operation was investigated by focusing on the operators’ hand motion. To be more precise, the discrete movement of hand was investigated using Fittsʹ law [41]. Fittsʹ law is an experimental formula concerning the time and distance of a voluntary hand motion. This law is valid for a reaching action of other body parts and of non‐ straight line courses [42]. Fittsʹ law has been used for the evaluation of computer interface designs, such as a GUI [43,44] or a stylus [45]. The present authors also guessed that the level of planning skill might be detected by checking the error against Fittsʹ law when inadequate action like a microslip occurred. Therefore, the travelling time/distance of the discrete hand motions in a machine manipulation on the console desk were investigated. Fig. 13 shows the layout of the switches on the control desk. Each planar coordinate value (x, y) of the switch position was registered as a table and the distance D was computed as a Euclid norm between the starting point (x,
y)s and the ending point (x, y)e of each hand reaching motion. The target width W is the size of lever grips and was also obtained from a pre‐registered table.
Figure 13. Layout of console switches and coordinate system The following formula was used for the analysis by referring the Fitts’ law.
RT k k x e 1 2 (6) x : log (2D / W) 2 , (7) where RT is the travelling time of one reaching action and k1 and k2 are constant coefficients. x is an index of difficulty and it depends only on geometric data of the distance and width. e is a fitting error to Fittsʹ law. e was
newly introduced in our study and was assumed to vary by cognitive load on the operator. The coefficients k1 and k2 were identified by a least square method with Eq. (6) using the multiple paired values { T,x}R from all the reaching actions data. Describing the identified coefficients as ˆk and1 ˆk ,2 the fitting error e[i] of the ith reaching action was computed as
e[i]RT[i] (k ˆ1k x[i])ˆ2 . (8) Fig. 14(a) and 14(b) show the mean value and covariance of the fitting error to Fittsʹ law, respectively. Each dotted line denotes the data of each participant and the bold solid line is the average for all eight participants. It was confirmed that the covariance of the fitting error decreased (Fig. 14(b)) as the number of trials increased, although the mean of e was almost flat (Fig. 14(a)). This fact indicates that covariance of the fitting error to Fittsʹ law is usable for evaluating the planning skill [46].
Figure 14. Transition of the fitting error to Fittsʹ law: (a) mean, (b) covariance
D2. Estimator of operatorʹs intention
Since planning is a decision of operational intentions, a function to guess the operatorʹs intention is also required for an intelligent assistive machine, which enhances human operation. Estimation of a userʹs intention is quite useful for various applications, such as in assistance software [47], prediction of usersʹ requests on the Internet [48] and marketing [49]. Estimation of intentions for a general human‐machine system is, however, more difficult than the above‐mentioned successful examples since it is also difficult to identify the information types which are utilized for the operatorʹs decision making.
To address this problem, the present authors proposed an algorithm to estimate an operatorʹs intention using the clustering technique with the multivariate time‐sequence data of the machine operation [50]. In the algorithm, the intentions are assumed to be an internal status of the human model. Therefore, the problem can be considered as a design of an observer that estimates the operatorʹs internal status in the human‐machine system, which consists of a human (operator), a machine (to be manipulated), an environment and the work task, as shown in Fig. 15.
Figure 15. System architecture of human‐machine system Describing these three types of statuses as the machine status (M‐status, qM), the environment status (E‐status, qE) and the task status (T‐status, qT), a human during machine operation can be defined as an information‐ processing system, H(z) s , wheres : [q ,q ,q ] TM TE T TT is the information to be recognized by the human, is the output of operation commands to the machine and z is the intention of the human. Furthermore, based on the concept of `spotlight of selective attentionʹ in Global Workspace Theory [51], mathematical expression of z is defined as a vector z [0,1] nzof which element
corresponds to one intention strength of one operation action (nz is a size of vector z). Then, the probabilistic distribution bel(z), that is a belief of the intention z, is estimated by the Bayes filtering technique. The basic algorithm for estimation of bel(z) is given as follows:
Algorithm: Bayes filter
bel(z )t
p(z |s ,z ) bel(z ) dzt t t 1 t 1 t 1 (9) bel(z )t p( |z ) bel(z )t t t , (10) where subscript t and t‐1 mean the time counters. This algorithm is defined with iterative equations that are computed from time index t=1 to the final time T.t t t 1
p(z |s ,z ) corresponds to a probabilistic distribution of a transition of intention from zt‐1 to zt given input st.
t t
p( |z ) is the conditional probabilistic distribution of judgment that outputs t if the intention zt happens to be true. is a so‐called Bayes normalization constant. Eq. (9) is a prediction to obtain a belief bel(z ) att the time of t. Eq. (10) is called a measurement update and adjusts the prediction bel(z ) byt considering the probabilityp( |z )t t . Via this update, a new belief bel(z ) att time t is obtained. The estimation of the intention, say ˆz ,t is computed as a median of the distribution ofbel(z ) .t
In our approach, the state transition relation of an operatorʹs intentions was formed using Self‐Organizing Map (SOM), which was trained using the measured data of the operation and environmental variables with the reference intention sequence. SOM is an artificial neural network for discriminating multi‐dimensional data and is adequate for data processing of large dimensional data since the SOM technique can compress multi‐ dimensional information into a lower (two) dimensional map by keeping original topological information. In
other words, similar types of data gather together and different types get away from each other on the clustering map. Such Bayesian property embedded in the SOM is expected to be utilized to predict a transition of status and we utilized it for the Bayes filtering algorithm to compute the probability of state transition. Refer to [50] for the detail.
Applying the proposed algorithm to the rr‐task, the estimatorʹs ability was verified. First, types of remote operation of the construction equipment and the definitions of elements (1)z,(15)z in intention vectors z were defined, as summarized in Table 2.
Table 2. Classified operational intention modes, variables and labels
Second, the SOM was trained using time‐sequence data including the machineʹs status and command of operation. The obtained SOM is shown in Fig. 16. Labels described in Table 2 are attached to each cluster of the corresponding mode in the figure. It can be confirmed that obvious clusters are formed.
Figure 16. Operation modes map generated by SOM
Third, using the above‐mentioned Bayes filtering algorithm with the obtained SOM, say the SOM‐Bayes estimator, intentions of other operators were estimated as a validation. Fig. 17 shows the transitions of the estimated intention by comparing with the human‐ discerned intentions. Lines show the transitions of intention modes from T/A‐a ((1)z ) to T/TU ((15)z ). Red lines were identified by the SOM‐Bayes estimator. Blue lines were discerned by a human analyst. The red lines by the estimator overlap with the blue lines by the human analyst well in the case of almost all intention modes.
Figure 17. Transitions of intentions (Red lines: by SOM‐Bayes
estimator, Blue lines: by human analyst)
Additional analysis confirmed that the estimator could identify the intention modes at 44‐94% concordance ratios against normal intention modes whose periods can be found by about 70% of human analysts [50]. Moreover, it was shown that distances between clusters on the SOM have a strong relation to the operatorʹs skill level in task performance [52]. As shown above, it was demonstrated that not only estimation of intentions but also evaluation of skill is available by utilizing SOM. In addition, the SOM‐Bayes intention estimator can be enhanced by combination with a Petri net based on ontology, which was shown in the previous Section III.3. Refer to [53] for the details.
4. Comprehensive analysis of high‐level skill
Skills from the sensor‐motor skill (S5) to the planning skill (S2) concern a relationship between one operator and a machine(s). Social skill (S1) differs from those skills in terms of an existence of more than one person; hence, the social skill is conceptually highly independent compared with the other skills. Social skill is, however, inseparably connected to the other skills since an enhancement of social skill comes after mastering those other skills. For instance, the planning skill is often taught to other people via social communication.
The authors studied social skill by using the cc‐task investigating other skills. The cc‐task is a type of the vehicle tele‐operation task, which is similar to dd‐task. In this experiment, the participants joined the cc‐task after they had mastered the manipulation of a virtual vehicle so as to eliminate an effect of a learning process of the sensory‐motor skill. Several studies about social skill using the cc‐task are introduced below.
A. Role in a collaborative work
Individual characteristics of the operatorsʹ behaviour in the collaborative work were investigated through analysis of log data, including the machine operation and vehicle motion of all participants. The individual characteristics were classified into three types: observer, leader and transporter. Through analysis, it was confirmed that adequate and obvious role allotment according to circumstance increased the efficiency of the whole work [54]. Fig. 18 shows that the distribution between the ratio of conveying and the ratio of observation had spread as the cooperative skill increased. This study could present one of indexes to evaluate social skill.
Figure 18. Change of ratio of action and observation in the
cooperative work.
B. Analysis of spoken language
From the point of view of communication, which is a representative property of social skill, corpuses of utterance in conversation among participants were analysed. It was found that corpuses used among them were commonalized as the social skill increased and that common ground was formed [55]. This fact indicates that sharing of a common communication scheme specialized for a given task is significant for a skilled team.
C. Analysis of eye motion
Eye motion is important not only for visuomotor control and perception to recognize circumstance but also for nonverbal communication. Eye motion occurs in order to guess another personʹs intention by watching the eyes, or is utilized as a cue for mutual confirmation with other people. A well‐known example is a joint attention [56]. Direction of gaze is also utilized for nonverbal communication by combination with pragmatics [57]. These follow that eye motion may reflect social skill. Therefore, in this subsection, an attempt to derive an equation to estimate team skill from the measurement of the eye motion of an individual is shown.
Basic eye‐movements are saccade, fixation, smooth pursuit and nystagmus. In our study, saccade and fixation were investigated in common with existing studies [58, 59]. Specifically, the following factors were investigated: (1) response time of eye‐movement against the target object’s motion, (2) gaze duration, (3) frequency
of saccade motion, (4) velocity of saccade and (5)
movement distance of saccade. Moreover, it is known
that the driver of the vehicle tends not to notice events
captured in the peripheral vision area since he/she
concentrates on the direction of travel of the vehicle [60].
It is also known that inessential eye movement of a
skilled driver decreases [61]. Taking into consideration
these facts, two cases of low and high speed driving were
considered against the above‐mentioned five factors and
total ten characteristics factors were investigated in our
study.
Since the aim of the analysis was to induce an equation,
which evaluates a canonical team‐skill, the induction was
processed eliminating outliers from the measured data.
Specifically, data including strong individual differences
were eliminated by the Smirnov‐Grubbsʹ outlier test
(p<0.1) and the normality of the remaining data was
checked by Lilliefors test (p<0.1). Then, correlation of
averages of each factor among participants was
investigated. It was confirmed that three factors, which
are saccade velocity of high‐speed driving, saccade
distance of high‐speed driving and saccade velocity of
low‐speed driving, have a high correlation with the
number of trials. Next, these three eye‐movement factors
were decomposed by PCA. Finally, an equation to
estimate a team skill was derived by using the weighting
coefficients corresponding to the first dominant
component vector on the PCA. Transition of the skill
values predicted by the deduced equation is shown in
Fig. 19. Investigating those predicted values against
another skill index, a high correlation (0.56‐0.84) was
confirmed. Hence, the effectiveness of the deduced skill
equation was confirmed [62]. This result can be utilized to
evaluate team skill in collaboration work using the
machine manipulation.
Figure 19. Transition of predicted skill level (Seven participantsʹ and its average data)
5. Conclusion
This paper introduced a hierarchical classification of
skills, which is helpful for designing assistive systems for
machine operators. After the experimental tasks for
investigations into the five types of skills in the
classification were explained, research applications and
the outcomes, which are the skill evaluation methods and
assistive system designs, were presented. Although any
task is related to various types of skill level, utilization of
the hierarchical classification of skills will make what is
important for designing an adequate assistive system for
the given task clear.
6. Acknowledgments
This research was supported by a Grant‐in‐Aid for the
21st Century COE (Centre of Excellence) Programme and
a Grant‐in‐Aid for Scientific Research (A) from the
Ministry of Education, Culture, Sports, Science and
Technology, Japan. The authors wish to thank the
Ministry for their generous financial assistance and the
many participants who cooperated for our experiments.
7. References
[1] K. Furuta, “Control of Pendulum: From Super
Mechano‐System to Human Adaptive Mechatronics”
Plenary, Proceedings of the 42nd IEEE Conference on
Decision and Control, Hawaii, USA, 2003: 1498– 1507.
[2] F. Harashima, “Human Adaptive Mechatronics”
Keynote speech, Proceedings of the IEEE Industrial
Electronics Society, Raleigh, USA, 2005.
[3] F. Harashima and S. Suzuki, “Intelligent Mechatronics
and Robotics” keynote speech, Proceedings of the
2008 IEEE International Conference on Emerging
Technologies and Factory Automation (ETFA2008),
Hamburg, Germany, Sept. 2008.
[4] S. Suzuki, K. Kurihara, K. Furuta and F. Harashima,
“Assistance Control on a Haptic System for Human
Adaptive Mechatronics” Journal of Advanced
Robotics, 2006; 20(3): 323–348.
[5] S. Suzuki, K. Kurihara, K. Furuta, F. Harashima and Y.
Pan, “Variable Dynamic Assist Control on Haptic
Systems for Human Adaptive Mechatronics”
Proceedings of the 44th IEEE Conference on Decision
and Control and European Control Conference (CDC‐
ECCʹ05), Seville, Spain, December, 2005: 4596‐4601.
[6] S. Suzuki and K. Furuta, “Adaptive impedance control
to enhance human skill on a haptic interface system”
in the special issue of Adaptive Control Theory and
Applications, Journal of Control Science and
Engineering, 2012, Article ID 365067.
[7] K. Furuta, Y. Kado, S. Shiratori and S. Suzuki,
“Assisting control for pendulum‐like juggling in
human adaptive mechatronics” Journal of Systems
and Control Engineering, Special Issues, IMechE,
2011; 225(6): 709–720.
[8] J. Åkesson and K. J. Åström, “Safe manual control of
the Furuta pendulum” Proceedings of IEEE
International Conference Control Applications,
[9] F. Miyawaki, K. Masamune, S. Suzuki, K. Yoshimitsu and J. Vain, “Scrub nurse robot system – Intraoperative motion analysis of a scrub nurse and timed‐autobata‐based model for surgery” IEEE Transaction Industrial Electronics, 2005; 52(5): 1227‐ 1235.
[10] F. Harashima and S. Suzuki, “Human Adaptive Mechatronics and Human‐System Modeling” Plenary talk, Proceedings of the 12th IEEE International Workshop on Advanced Motion Control (AMC), Sarajevo, Bosnia and Herzegovina, 2012.
[11] S. Suzuki, “Human Adaptive Mecathorincs” IEEE Industrial Electronics Magazine, 2010; 4(2): 28‐35. [12] P. Foo, J. A. S. Kelso and G. C. de Guzman,
“Functional stabilization of fixed points: Human pole balancing using time to balance information” Journal of Experimental Psychology: Human Perception and Performance, 2000; 26: 1281‐1297.
[13] J. Åkesson and K. J. Åström, “Manual Control and Stabilization of an Inverted Pendulum” Proceedings of 16th IFAC World Congress, Prague, Czech Republic, 2005.
[14] J. L. Cabrera and J. G. Milton, “Stick balancing: On‐ off intermittency and survival times” Nonlinear Studies, 2003; 10(2): 1‐13.
[15] J. L. Cabrera and J. G. Milton, “Human stick balancing: Tuning Levy flights to improve balancing control” Chaos, 2004; 14(3): 691‐698.
[16] J. L. Cabrera and J. G. Milton, “On‐Off Intermittency in a Human Balancing Task” Physical Review Letters, 2002; 89(15): 158702/1‐4.
[17] A. Priplata, J. Niemi, M. Salen, J. Harry, L. A. Lipsitz and J. J. Collins, “Noise‐enhanced human balance control” Physical Review Letters, 2002; 89(23): 238101:1‐4.
[18] J. L. Cabrera, R. Bormann, C. Eurich, T. Ohira and J. Milton, “State‐dependent noise and human balance control” Fluctuations and Noise Letters, 2004; 4: L107‐L118.
[19] J. G. Milton, T. Ohira, J. L. Cabrera, R. M. Fraiser, J. B. Gyorffy, F. K. Ruiz, M. A. Strauss, E. C. Balch, P. J. Marin and J. L. Alexander, “Balancing with Vibration: A Prelude for `Drift and Act’ Balance Control” PLoS ONE, 2009; 4(10): e7427.
[20] S. C. Venkataramani, T. M. Antonsen Jr., E. Ott and J. C. Sommerer, “On‐off intermittency: Power spectrum and fractal properties of time series” Physica D, 1996; 96: 66‐99.
[21] J. Daunizeau, C. Grova, G. Marrelec, J. Mattout, S. Jbabdi, M. Peegrini‐Issac, J. M. Lina and H. Benali, “Symmetrical event‐related EEG/fMRI information fusion in a variational Bayesian framework” Neuroimage, 2007; 36(1): 69‐87.
[22] M. J. Herrmann, A.‐C. Ehlis and A.J. Fallgatter, “Frontal activation during a verbal‐fluency task as
measured by near‐infrared spectroscopy” Brain Research Bulletin, 2003; 61: 51‐56.
[23] Y. Noguchi, T. Takeuchi and K. L. Sakai, “Lateralized activation in the inferior frontal cortex during syntactic processing: An event‐related optical topography study” Human Brain Mapping, 2002; 17: 89‐99.
[24] M. Pena, A. Maki, D. Kovačiċ, G. Dehaene‐Lambertz, H. Koizumi, F. Bouquet and J. Mehler, “Sounds and silence: An optical topography study of language recognition at birth” Proceedings of the National Academy of Science of the USA, 2003; 100(20): 11702‐ 11705.
[25] S. Suzuki and F. Harashima, “Analysis of machine operation skills using hand discrete movement” Proceedings of the 2008 IEEE International Conference on Emerging Technologies and Factory Automation (ETFA2008), Hamburg, Germany, 2008: 156‐163.
[26] A. Tustin, “The Nature of the Operatorʹs Response in Manual Control and its Implications for Controller Design” Journal of Inst. of Electrical Engineering, 1947; 94(IIA): 190‐202.
[27] J. R. Ragazzini, “Engineering Aspects of the Human Being as a Servo‐Mechanism” Meeting of the American Psychological Association, 1948.
[28] S. Baron, D. L. Kleinman and W. H. Levison, “An Optimal Control Model of Human Response Part II: Prediction of Human Performance in a Complex Task” Automatica, 1970; 6: 371‐383.
[29] R. C. Miall and D. M. Wolpert, “Forward Models for Physiological Motor Control” Neural Networks, 1996; 9(8): 1265‐1279.
[30] R. C. Miall, D. J. Weir, D. M. Wolpert and J. F. Stein, “Is the Cerebellum a Smith Predictor?” Journal of Motor Behavior, 1993; 25(3): 203‐216.
[31] B. Mehata and S. Schaal, “Forward Models in Visuomotor Control” Journal of Neurophysiology, 2001; 88: 942‐953.
[32] D. M. Wolpert, R. C. Miall and M. Kawato, “Internal models in the cerebellum” Trends in Cognitive Sciences, 1998; 2(9): 338‐347.
[33] N. Bhushan and R. Shadmehr, “Computational nature of human adaptive control during learning of reaching movements in force fields” Biological Cybernetics, 1999; 81: 39‐60.
[34] S. Suzuki, F. Harashima and K. Furuta, “Human Control Law and Brain Activity of Voluntary Motion by Utilizing a Balancing Task with an Inverted Pendulum” Advances in Human‐Computer Interaction, 2010; Vol. 2010, Article ID 215825; doi:10.1155/2010/215825.
[35] M. A. Brackstone, B. Sultan and M. McDonald, “Findings on the approach process between Vehicles” Transportation Research Record, 2000; Vol.1724, paper No.00‐0831: 21‐28.
[36] H. Kobayashi, “Relations between eye gaze and cognitive factors in a transportation task using remote control” Proceedings of 18th IEEE International Symposium on Robot and Human Interactive Communication (RO‐MAN09), 2009, Toyama, Japan.
[37] R. Mizoguchi, M. Ikeda, K. Seta and J. Vanwelkenhuysen, “Ontology for Modeling the World from Problem Solving Perspectives” Proceedings of the International Joint Conference on Artificial Intelligence (IJCAI‐95) Workshop on Basic Ontological Issues in Knowledge Sharing, 1995: 1‐12. [38] E. S. Reed and D. Schoenherr, “The neuropsychology
of everyday life: On the nature and significance of micro‐slips in everyday activities” Unpublished manuscript, 1992.
[39] K. Suzuki, H. Mishima and M. Sasaki, “Affordance and Variability of Action ‐Environment, Action and Microslips‐” Journal of Japan Society for Fuzzy Theory and Systems, 1997; 9(6): 826‐837.
[40] H. Kobayashi, T. Yasuda and S. Suzuki, “The relations between development of humanʹs manipulative skills and physiological signals of brain and hand” Proceedings of the International Journal of Assistive Robotics and Mechatronics, 2006; 7(1): 49‐61.
[41] P. M. Fitts, “The information capacity of the human motor system in controlling the amplitude of movement” Journal of Experimental Psychology, 1954; 47(6): 381‐391.
[42] J. Accot and S. Zhai, “Beyond Fitts’ Law: Models for Trajectory‐Based HCI Tasks” in Proceedings of the SIGCHI conference on Human factors in computing systems (CHIʹ97), New York, USA, 1997: 295‐302. [43] M. J. McGuffin and R. Balakrishnan, “Fitts’ law and
expanding targets: Experimental studies and designs for user interfaces” Transactions on Computer‐ Human Interaction, 2005; 12(4): 388‐422.
[44] S. Zhai, S. Conversy, M. Beaudouin‐Lafon and Y. Guiard, “Human on‐line response to target expansion” in Proceedings of the SIGCHI conference on Human factors in computing systems (CHIʹ03), 2003; 5(1): 177‐184.
[45] X. Ren and S. Mizobuchi, “Investigating the Usability of the Stylus Pen on Handheld Devices” in Proceedings of The Fourth Annual Workshop on HCI Research in MIS, Las Vegas, USA, 2005: 30‐34. [46] S. Suzuki and F. Harashima, “Skill evaluation from
observation of discrete hand movements during console operation” Journal of Robotics, 2010; Vol. 2010, Article ID 967379: doi:10.1155/2010/967379. [47] E. Horvitz, J. Breese, D. Heckerman, D. Hovel and K.
Rommelse, “The Lumiere Project: Bayesian User Modeling for Inferring the Goals and Needs of Software Users” Proceedings of the Fourteenth Conference on Uncertainty in Artificial Intelligence, Madison, WI, USA, 1998: 256‐265.
[48] I. Zukerman, D. W. Albrecht and A. E. Nicholson, “Predicting Usersʹ Requests on the WWW” Proceedings of the Seventh International Conference on User Modeling, Banff, Canada, 1999: 275‐284. [49] J. Gutman, “A means‐end chain model based on
consumer categorization process” Journal of Marketing, 1982; 46: 60‐72.
[50] S. Suzuki and F. Harashima, “Estimation algorithm of machine operational intention by Bayes filtering with Self‐Organizing Map” Advances in Human‐ Computer Interaction, 2012; Vol.2011, Article ID 724587: doi:10.1155/2012/724587.
[51] B. J. Baars, A cognitive theory of consciousness, Cambridge University Press, New York, 1988. [52] S. Suzuki and F. Harashima, “Estimation of
operational intentions utilizing Self‐Organizing Map with Bayes filtering” Proceedings of the 2010 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS2010), Taipei, Taiwan, 2010: 2249‐2255. [53] S. Suzuki and F. Harashima, “Context‐aware
Bayesian intention estimator using Self‐Organizing Map and Petri net” Proceedings of 20th IEEE International Symposium on Robot and Human Interactive Communication (RO‐MAN2011), Atlanta, GA, USA, 2011: 314‐320.
[54] H. Igarashi, S. Suzuki, H. Kobayashi and F. Harashima, “Role Sharing Analysis on Multi‐ operator Cooperative Work” Proceedings of 18th IEEE International Symposium on Robot and Human Interactive Communication (RO‐MAN09), Toyama, Japan, 2009: 1060‐1065.
[55] S. Nakata, H. Kobayashi, M. Kumata and S. Suzuki, “Human Speech Ontology Changes in Virtual Collaborative Work” Proceedings of the 4th International Conference on Human System Interaction (HSI 2011), Yokohama, Japan, 2011: 363‐368.
[56] T. Yasuda, H. Kobayashi, “Influence of 10 Seconds Interval in Pragmatic Interpretation” Proceedings of 19th IEEE International Symposium on Robot and Human Interactive Communication (RO‐MAN10), Viareggio, Italy, 2010.
[57] T. Yasuda and H. Kobayashi, “Timing of adults’ utterances and interpretation of word meanings in a discrepant labeling situation” Proceedings of the 30th Annual Meeting of the Cognitive Science Society, 2008: 675–680.
[58] T. Ito, H. Matsubara and R. Grimbergen, “Cognitive Science Approach to Shogi Playing Processes (1) ‐ Some Results on Memory Experiments” IPSJ; 43(10): 2998‐3011.
[59] K. Sakita, K. Ogawara, S. Murakami, K. Kawamura and K. Ikeuchi, “Flexible Cooperation between Human and Robot by interpreting Human Intention from Gaze Information” Proceedings of IEEE/RSJ International Conference on Intelligent Robotics and Systems (IROS2004), Sendai, Japan, 2004: 846‐851.
[60] T. Miura and K. Shinohara, “Safety Problems of the
Car Navigation from the Viewpoint of Psychology of
Attention” IATSS Review, 2001; 26(4): 33‐44.
[61] T. Ohno and H. Ogasawara, “Information Acquisition
Model of Highly Interactive Tasks” Proceedings of
ICCS/JCSS’99，Tokyo, Japan, 1999: 288‐293.
[62] K. Kuronuma, S. Suzuki and H. Igarashi, “Estimation
method of awareness level using an iconic model for
support robots” Proceedings of IFToMM Asian
Conference on Mechanism and Machine
Science,Taipei, Taiwan, 2010.
