IDEAS is not in itself a theoretically based approach. Rather, it is an application framework, which allows the insights from a range of theoretical positions or pragmatic research to be cast into the form of a simple linear additive model. This model allows the development of a CMWL index as a function of the various factors that emerge from the theoretical and experimental research that has been carried out in this domain.
The IDEAS approach provides a promising approach to CMWL evaluation from the following perspectives.
• Flexibility: the modelling approach allows a wide variety of factors that might affect CMWL to be included in the construction of a predictive model of CMWL.
• Breadth of modelling: The model developed in the IDEAS environment can include both ‘soft factor’ such as subjective opinions and ‘hard data’ e.g. length of shift and work break schedules to be included.
• Predictive capability: Although several of the techniques considered in this review could potentially provide predictions of CMWL, only IDEAS provides the capability to make these predictions as a function of the context i.e. characteristics of the task, the individual and the environment.
• Linkage with error prediction: As discussed earlier, the IDEAS approach can evaluate the contribution of CMWL as part of a larger
predictive model of human error. This means that concepts of ‘acceptable workload’ at both the low and high demand ends of the spectrum can be defined operationally in terms of failure probabilities for the tasks being assessed. This is particularly valuable if these measures are to be used as part of an overall risk management framework. It would be possible for example to decide whether a given expenditure to produce a desired CMWL was defensible in terms of the expected levels of risk reduction. (i.e. reducing the likelihood of a particular undesirable outcome).
6. Industry Specific Workload Assessment
Tools
This section provides a listing of a selection of the main workload assessment tools which have been developed for use in specific contexts. For each tool several items are noted (where they are relevant or known): Tool name, originating industry sector, sponsor organisation, theory base and core measurement methodology. This information will enable the reader to follow through on any particular tool of interest. Because the most important tools have already been discussed at length elsewhere in this report the list of assessment tools are presented without further commentary.
Tool Originating Industry Sector
Sponsor Theory Base Core
Methodology
ATCO-TAD Air Traffic
Control (ATC) CAA Generic IP Task load
AWAS ATC EuroControl MRT Subjective
Assessment
Bedford Scale Aviation RAE SRT Subjective
Assessment BLV
Questionnaire Vehicle handling NATO Generic IP Assessment Subjective
CART Aviation AFRL
CrewCut / WinCrew
Defence ARL MRT Task load
ECG response Cardiovascular Physiological
EEG response Neurology Physiological
Electrodermal
response Neurology Physiological
EOG Visual
Neurology Physiological
FWTCI Naval Ops NAWC Task load
IMPRINT Defence (Land
Ops) ARL MRT Task load
IPME Manual Ops MA&D Subjective
Assessment Lysaght Test
Battery Vehicle handling SCH Secondary Task
M-Model Obersver Pro
Generic Commercial
Noldus Behavioural Primary Task
MAN-SEVAL Defence (Land
Ops) ARL MRT Task load
Tool
Originating Industry
Sector Sponsor Theory Base
Core
Methodology
Modified Cooper-Harper
Aviation Generic IP Subjective
Assessment NASA TLX Aerospace NASA Generic IP Subjective
Assessment OFM-COG Maritime
Transportation University R&D Generic IP Cognitive Task Analysis
OWLKNEST Military ARI-IFRU Generic IP IKBS
POP Manual Ops MA&D Subjective
Assessment
PUMA ATC NATS MRT Task load
Qh Air/Rail
Transportation HEL MRT Task load
Rummel Test Battery Vehicle handling NATO SCH Secondary task Sequential Judgement Scale Vehicle
handling MRT Subjective Assessment
SHIPSHAPE Generic CIL IKBS
SOLE Military DRDC Generic IP Task load
STRES Aviation AFRL Generic IP Secondary
Task
SWAT Aerospace Armstrong
Aerospace Generic IP Assessment Subjective
SWORD Aviation ARL MRT Subjective
Assessment TAWL Defence
(Aviation)
LHX VACP-IP Cognitive Task
Analysis TDMW
(timeline) Rail RSSB Generic IP Timeline
WCFIELDE Military DRDC MRT Task Load
Figure 37 Industry specific workload assessment tools RAE Royal Aircraft Establishment
CART Combat Automation Requirements Testbed CAA UK Civil Aviation Authority
RSSB Rail Standards and Safety Board ARL US Army Research Laboratory AFRL Air Force Research Laboratory
NATS National Air Traffic System NAWC Naval Air Warfare Center
7. Conclusions
The review of the scientific literature on workload assessment reveals that the topic has proved an extremely rich, vibrant and productive work area with investigators contributing to the creation of an extremely large body of knowledge. Mental workload studies have tended to be of two types: Pure science - where investigators have a special interest in the development and evolution of cognitive theory and applied work – where the objective has been to measure the impact of task demands on the performance of the human operator in work settings. With some important exceptions, earlier work tended to reflect pure science concerns. Later investigations have tended to reflect applied concern with the role played by unsatisfactory workload levels in work based accidents. Within this latter strand of work, the literature further sub-divides into two sub-areas. One important area of work has concentrated on investigating human performance in conditions of high workload. Here the objective is mainly to identify psychological and environmental stressors that reduce a person’s ability to complete a task or range of tasks. The second work area has concentrated on conditions of low workload, where the operator must remain alert to low frequency but highly critical events. This latter work stream deals mainly with the maintenance of watchkeeping tasks and there is particular interest in mapping the vigilance decrement errors that occur during target detection tasks.
Given the sheer size of the literature, it is difficult to provide a summary statement that encapsulates each individual approach. However, within applied work, it seems clear that each practitioner takes up a position in relation to the topic according to three parameters: background theory, preferred workload measure, and target industry.
There are four main theoretical positions with regard to the topic of mental workload. Three of these, the single channel hypothesis, single resource theory and multiple resource theory, are rooted in the view of the human as a processor of information - much in the vein of a computer system. The fourth (e.g., EPIC) provides a composite model rooted in cognitive science. Conclusive evidence to allow investigators to choose between each theory has not yet been forthcoming but, on balance, it would seem fair to conclude that multiple resource theoretic models have been most successful in accounting for benchmark task interference affects. Consequently, most assessment tools tend to be based on the MRT perspective. This theory proposes that cognitive processes are limited both in terms of resources and cognitive structures and that the likelihood the dual performance will be affected by task demands depends largely on the types of tasks to be performed.
With regard to workload measures, the review confirms that four different types of measures have been deployed in the assessment of mental
Consideration of the development of industry specific tools reveals that most industries involving a significant loss potential have invested in the creation of workload assessment tools. Perhaps the interesting point here is the low levels of tool migration and sharing of data between sectors. Each industry has preferred to develop its own tool possibly due to a feeling that it has special requirements. For example, tools developed for use in assessments involving road vehicles have tended to favour measures obtained from the performance of primary or secondary tasks, presumably because these methods enable collection of workload data concurrent with performance of the task in a real-world driving situation.
References
Allender, L., Salvi, L. Promisel, D. (1997) Evaluation of Human Performance under Diverse Conditions via Modeling Technology. Proceedings of Workshop on Emerging Technologies in Human Engineering Testing and Evaluation, NATO Research Study Group 24, Brussels, Belgium.
Allport, A. (1993). Attention and control: Have we been asking the wrong questions? A critical review of twenty five years. In D.E. Meyer and S. Kornblum, eds., Attention and Performance XIV. Cambridge, MA: MIT Press Andreassi, J.L. (2000). Psychophysiology: Human behaviour and physiological response (4th edition). Lawrence Erlbaum Associates.
Beatty, J. (1982). Task-evoked pupillary responses, processing load, and the structure of processing resources. Psychological Bulletin, 91, 276-292.
Beevis, D. (1992). Analysis techniques for man-machine systems design. Report AC/243 (Panel 8) TR/7 Vol. 2. Brussels: NATO Defence Research Group.
Beevis, D., R. Bost, B. Döring, E. Nordø, J.-P. Papin, I.H. Schuffel, and D. Streets. (1994). Analysis Techniques for Man-Machine System Design. AC/243 (Panel 8) TR/7. Defence Research Group, North Atlantic Treaty Organization.
Biers, D.W. and Masline, P.J. (1987). Alternative approaches to analyzing SWAT data.. In Proceedings of theHuman Factors Society 31st Annual Meeting (pp. 63-66). Santa Monica, CA: Human Factors and Ergonomics Society.
Bornemann, E. (1942). Untersuchungen über den Grad der geistigen Beanspruchung. II. Teil: Praktische Ergebnisse. Arbeitsphysiologie, 142, 12, 173-191 (cf: NATO, 2001).
Boucsein, W., (2005). Electrodermal Measurement. In N.A.Stanton, A.Hedge., K.Brookhuis, E. Salas., H.Hendrick (eds) Handbook of Human Factors and Ergonomic Methods. CRC Press: London.
Boyd, S. (1983). Assessing the validity of SWAT as a workload measurement instrument, Proceedings of the Human Factors and Ergonomics Society 27th Annual Meeting (pp. 124-128). Santa Monica, CA: Human Factors Society. Broadbent, D.E. (1958). Perception and communication. London: Pergamon. Brown, I.D. & Poulton, E.C. (1961). Measuring the spare ‘mental’ capacity of cardrivers by a subsidiary task. Ergonomics, 4, 35-40.
Caggiano, D.M., & Parasuraman, R., (2004). The role of memory representation in the vigilance decrement. Psychonomic Bulletin and Review. 11 (5), pp. 932-937.
Cooper, GE & Harper, RP (1969). The use of pilot ratings in the evaluation of aircraft handling characteristics. Washington, DC: NASA, Report No. TN-D- 5153.
Corcoran, D.W.J., Mullin, J., Rainey, M.T., (1977). The effects of raised signal and noise amplitude during the course of vigilance tasks. In R.R.Mackie, (ed) Vigilance: Theory, Operational Performance and Physiological Correlates. Plenum: New York.
Corwin, W.H., Sandry-Garza, D.L., Biferno, M.H., Boucek, G.P., Logan, A.L., Jonsson, J.E., and Metalis S.A. (1989). Assessment of crew workload measurement; methods, techniques, and procedures: Volume 1. Process, methods, and results (Tech. Report WRDC-TR-897006). Wright-Patterson Air Force Base, OH: Wright Research and Development Center, Air Force Systems Command.
Damos, D.L. (Ed.) (1991). Multinle task performance. Washington, DC: Taylor Davies, D.R., & Parasuraman, R., (1982). The Psychology of Vigilance. Academic Press: London.
de Waard, D., (1996). The Measurement of Drivers’ Mental Workload. PhD Thesis. The Traffic Research Centre VSC, University of Groningen, The Netherlands.
DiDimenico, A.T., (2003). An investigation on subjective assessments of workload and postural stability under conditions of joint mental and physical demands. PhD Thesis, Virginia Polytechnic Institute and State University. Doll, T.J., & Hanna, T.E., (1989). Enhanced detection with bimodal sonar displays. Human Factors. 31: pp. 539-550.
Fisk, A.D., & Schneider, W., (1981). Control and automatic processing during tasks requiring sustained attention: A new approach to vigilance. Human Factors. 23: pp.737-750.
Galinsky, T.L., Warm, J.S., Dember, W.N., Weiler, E.M., & Scerbo, M.W., (1990). Effects of event rate presentation of subjective workload in vigilance performance. Society for Philosophy and Psychology, Chicago, Illinois.
Geddie, J.C., Boer, L. C., Edwards, R. J., Enderwick, T. P., Graff, N. Pfendler, C. Ruisseau, J., van Loon, P. A. (2001) NATO Guidelines on Human Engineering Testing and Evaluation. RTO Technical Report 21 BP 25, Cedex: France
Griffin, J.A., Dember W.N., & Warm, J.S., (1986). Effects of depression on expectancy in sustained attention. Motivation and Emotion. 10: pp.195-205. H.R. Booher (Ed.), (1990) Manprint: An approach to systems integration. New York: Van Nostrand Reinhold.
Hancock, P. A., Meshkati, N., and Robertson, M. M. (1985). Physiological reflections of mental workload. Aviation, Space, and Environmental Medicine, 56(11), 1110-1114.
Hancock, P.A., and J.S. Warm. (1989). A dynamic model of stress and sustained attention . Human Factors 31:519-537.
Hart, S.G. & Staveland, L.E. (1988). Development of NASA-TLX (Task Load Index): results of empirical and theoretical research. In P.A. Hancock & N. Meshkati (Eds.), Human Mental Workload. Amsterdam: North-Holland.
Hart, S.G. and Wickens, C.D. (1990). Workload assessment and prediction. In H.R. Booher Ed.), MANPRINT. An approach to systems integration (pp. 257- 296). New York: Van Nostrand Reinhold.
Hendy, K.C., Hamilton, K.M., & Landry, L.N., (1993). Measuring subjective workload: When is one scale better than many? Human Factors. 35, pp. 579- 601.
Hill, S., Iavecchia, H., Byers, J., Bittner, A.C., Zaklad, A.L., and Christ, R.E., (1992). Comparison of four subjective workload rating scales. Human Factors, 34 (4), 429-439.
Hoffman, J.E., M.R. Houck, F.W. MacMillan, R.F. Simons, and J.C. Oatman (1985) Event-related potentials elicited by automatic targets: A dual-task analysis. Journal of Experimental Psychology: Human Perception and Performance 11:50-61.
Hugdahl, K., (2001). Psychophysiology: The Mind Body Perspective. Perspective in Cognitive Neuroscience. Harvard University Press. Canbridge: MA>
Humphrey, D.G. & Kramer, A.F. (1994). Toward a psychophysiological assessment of dynamic changes in mental workload. Human Factors, 36, 3- 26.
Israel, J.B., Chesney, G.L., and Donchin, E. (1980). The event related brain potential as an index of displaymonitoring workload. Human Factors, 22, 211- 224.
Jerison, H.J., (1963). On the decrement function in human vigilance. In D.N.Buckner & J.J.McGrath., Vigilance: A Symposium. McGraw Hill: New York.
Johannsen, G., Moray, N., Pew, R., Rasmussen, J., Sanders, A., and Wickens, C. (1979). Final report of experimental psychology group. In N. Moray (Ed.), Mental workload, its theory and measurement. New York. Plenum Press, 101-114.
Joshi, A., Dember, W.N., Warm, J.S., & Scerbo, M.W., (1985). Capacity demands in sustained attention. Psychonomic Society. Boston: Mass
Just, M. A. and Carpenter, P. A. (1993). The intensity dimension of thought: Pupillometric indices of sentence processing, Canadian Journal of Experimental Psychology 47: 310-339.
Kahneman, D. (1973). Attention and effort. Englewood Cliffs, NJ: Prentice Hall.
Kieras, D. E. and Meyer, D. E. (1997). An overview of the EPIC architecture for cognition and performance with application to human-computer interaction. Human-Computer Interaction, 12, 391-438.
Kramer, A.F. (1991). Physiological metrics of mental workload: a review of recent progress. In D.L. Damos (Ed.), Multiple-task performance. (pp. 279- 328). London: Taylor & Francis.
Lambertus, J.M., deWaard, D., & Brookhuis, K.A., (2005). Estimating mental effort using heart rate and heart rate variability. In N.A.Stanton, A.Hedge., K.Brookhuis, E. Salas., H.Hendrick (eds) Handbook of Human Factors and Ergonomic Methods. CRC Press: London.
Lanzetta, T.M., Dember, W.M., Warm, J.S., & Berch, D.B., (1987). Effects of task type and stimulus heterogeneity on the event rate function in sustained attention. Human Factors. 29: pp.625-633.
Lee, J.D., and Sanquist, T.F., (2000). Augmenting the Operator Function Model with Cognitive Operations: Assessing the Cognitive Demands of Technological Innovation in Ship Navigation. IEEE Transactions on Systems, Man and Cybernetics – Part A: Systems and Humans. 30, (3), pp. 273-285. Little, R., (1993) Crew Reduction in Armored Vehicles Ergonomic Study Report number: ARL-CR-80, U.S. Army Research Laboratory.
Loeb, M., Noonan, T.K., Ash, D.W., Holding., D.H., (1987) Limitations in the cognitive vigilance increment. Human Factors. 29: pp.661-674
Lysaght, R. J., Hill, S. G., Dick, A. O., Plamondon, B. D., Linton, P. M., Wierwille, .W., Zaklad, A. L., Bittner Jr, A. C., and Wherry, R. J. (1989). Operator workload: Comprehensive review and evaluation of workload
methodologies (ARI Technical Report 851). Alexandria, VA: U.S. Army Research Institute for the Behavioral and Social Sciences.
Mackworth, N.H., (1948). The breakdown of vigilance during prolonged visual search. Quarterly Journal of Experimental Psychology 1: pp 6-21.
Makeig, S., and T.P. Jung (1996) in Changes in alertness are a principal component of variance in the EEG spectrum, NeuroReport, 7(1), 213-216. Mallis, M.M., & Dinges, D.F., (2005) Monitoring Alertness by Eyelid Closure. In N.A.Stanton, A.Hedge., K.Brookhuis, E. Salas., H.Hendrick (eds) Handbook of Human Factors and Ergonomic Methods. CRC Press: London.
Meijman, T.F. & O’Hanlon, J.F. (1984). Workload. An introduction to psychological theories and measurement methods. In P.J.D. Drenth, H. Thierry, P.J. Willems & C.J. de Wolff (Eds.), Handbook of Work and Organizational Psychology. (pp. 257-288). New York: Wiley.
Miller, R.A., (1985) A systems approach to modeling discrete control performance, in W. B. Rouse (Ed.), Advances in Man-Machine Systems Research vol 11. Greenwich, CT: JAI, , pp. 177-248.
Mitchell, C. M. (1996). GT-MSOCC: Operator models, model-based displays, and intelligent aiding. In W. B. Rouse (Ed.), Human/technology interaction in complex systems (Vol. 8, pp. 67-172). Greenwich, CT: JAI Press Inc.
Mitchell, C.M., & Miller, R.A., (1986). A discrete control model of operator function: A methodology for information display design,” IEEE Trans. Syst., Man, Cybern., vol. SMC-16, pp. 343–357.
Mitchell, C.M., (1987) “GT-MSOCC: A research domain for modelling human- computer interaction and aiding decision making in supervisory control systems,” IEEE Trans. Syst. Man Cybern., vol. SMC-17, no. 4, pp. 553-570, July/Aug. 1987.
Mitchell, D. K. (2000) Mental Workload and ARL Workload Modeling Tools. Army Research Laboratory, Report No ARL-TN-161.
Moray, N. (1967). Where is attention limited? A survey and a model. Acta Psychologica, 27, 84-92.
Muckler, F.A. & Seven, S.A. (1992). Selecting performance measures: ‘objective’ versus ‘subjective’ measurement. Human Factors, 34, 441-455. Mulders, H., Meijman, T., Mulder, B., Kompier, M., Broersen, S., Westerink, B. & O’Hanlon, J. (1988). Occupational stress in city bus drivers. In J.A. Rothengatter & R.A. de Bruin (Eds.), Road user behaviour: theory and
NASA Task Load Index , (1986). TLX: Paper and pencil version. Moffet Field, CA: NASA-Ames Research Center, Aerospace Human Factors Research Division.
Nataupsky, M. and Abbott, T. S. (1987). Comparison of workload measures on computer-generated primary flight displays. In Proceedings of the Human Factors Society 31st Annual Meeting (pp. 548-552). Santa Monica, CA: Human Factors Society.
NATO (2001). Guidelines on Human Engineering: Testing and Evaluation. Final Report of the RTO Human Factors and Medicine Panel. Neuilly-Sur- Seine, CEDEX, France.
Navon, D. (1984). Resources - a theoretical soup stone? Psychological Review, 91, 216-234.
Navon, D. and Gopher, D. (1979). On the economy of the human processing system. Psychological Review, 86(3), 214-255.
Norman, D. and Bobrow, D. (1975). On data-limited and resource-limited processing. Journal of Cognitive Psychology, 7, 44-60.
Nygren, T.E. (1991). Psychometric properties of subjective workload measurement techniques: implications for their use in the assessment of perceived mental workload, Human Factors, 33, 17-33.
O’Donnell, R.D. & Eggemeier, F.T. (1986). Workload assessment methodology. In K.R. Boff, L. Kaufman & J.P. Thomas (Eds.), Handbook of perception and human performance. Volume II, cognitive processes and performance. (pp 42/1-42/49). New York: Wiley.
Pashler, H., (1994). Dual-task interference in simple tasks: data and theory. Psychology Bulletin. 116, 220– 244.
Penington, J., Joy, M., Kirwan, B., (1993) A staffing assessment for a local