Chapter 7 outlines the JAR-OPS requirements concerning the use of behavioural markers. This Appendix contains details of some of the behavioural marker schemes, both formal and informal, which are available. It should not be interpreted as a definitive list. The following schemes are addressed:
• NOTECHS
• University of Texas behavioural markers
• LMQCRM standards
Further examples of behavioural markers currently used by Operators may be found on www.raes-hfg.com on the CRMAP page.
1.2 Frequently asked questions concerning behavioural markers
The following text has been extracted verbatim from the GIHRE report (Group Interaction in High Risk Environments)
1 What are behavioural markers?
• Observable, non-technical behaviours that contribute to superior or substandard performance within a work environment (for example, as contributing factors enhancing safety or in accidents and incidents in aviation).
• Observable behaviours of teams or individuals.
• Usually structured into a set of categories.
• The categories contain sub-components that are labelled differently in various behavioural marker systems (e.g. NOTECHS: “elements“ and
“markers“ = UT (LOSA): “anchors“).
2 How are behavioural markers derived?
• From analysis of data from multiple sources regarding performance that contributes to successful and unsuccessful outcomes (e.g. accident investigation, confidential incident reporting systems, incident analysis, simulator studies, task analysis, interviews, surveys, focus groups, ethnographies).
3 What makes a good behavioural marker?
• It describes a specific, observable behaviour, not an attitude or personality trait, with clear definition (enactment of skills or knowledge is shown in behaviour).
• It has demonstrated a causal relationship to performance outcome.
• It does not have to be present in all situations.
• Its appropriateness depends on context.
• It uses domain specific language that reflects the operational environment.
• It employs simple phraseology.
• It describes a clear concept.
4 What are the domains of application?
Behavioural markers can be used in any domain where behaviour relating to job performance can be observed. However, they are expensive to develop and utilise given the level of training and calibration required for users. At present, they tend to be found in occupations where safety is prime and high fidelity simulators are used for training and assessment, e.g. in aviation, nuclear power generation, military settings, and, to a lesser degree, in medicine (anaesthesia and surgery), where simulation is less widely employed.
5 What are the uses of behavioural markers?
• To enable performance measurement for training and assessment, evaluation of training, safety management, and research.
• To highlight positive examples of performance.
• To provide a common vocabulary for training, briefing and debriefing, communication, regulation, research, and to connect different domains of safety (e.g. incident analysis and performance tracking).
• To build performance databases to identify norms and prioritise training needs.
• To compare sub-groups in organisations (e.g. aircraft fleets, etc.).
• To give feedback on performance at individual, team, organisational, and system levels.
• To establish co-operation between safety/quality, training, and operations.
6 What are characteristics of good behavioural marker systems?
• Validity: in relation to performance outcome.
• Reliability: inter-rater reliability, internal consistency.
• Sensitivity: in relation to levels of performance.
• Transparency: the observed understand the performance criteria against which they are being rated; availability of reliability and validity data.
• Usability: easy to train, simple framework, easy to understand, domain appropriate language, sensitive to rater workload, easy to observe.
• Can provide a focus for training goals and needs.
• Baselines for performance criteria are used appropriately for experience level of ratee (i.e. ab initio vs. experienced ratees).
• Minimal overlap between components.
7 What are the limitations of behavioural marker systems?
Cannot capture every aspect of performance and behaviour due to:
• Limited occurrence of some behaviours;
• These are important but infrequent behaviours, such as conflict resolution.
• Limitations of human observers – distraction, overload (e.g. in complex situations, large teams).
8 What considerations must be made when using a behavioural marker system?
• Raters require extensive training (initial and recurrent) and calibration.
• Behavioural marker systems do not transfer across domains and cultures without adaptation (e.g. western markers in eastern cultures, or from aviation to medicine).
• Behavioural marker systems need proper implementation into an organisation, and need management and workforce support.
• phased introduction of behavioural marker systems required to build confidence and expertise in raters and ratees.
• Application of the behavioural marker system must be sensitive to the stage of professional development of the individual, and to the maturity of the organisational and professional culture (e.g. whether used as a diagnostic, training, and/or assessment tool).
• Use must consider context (e.g. crew experience, workload, operating environment, operational complexity).
9 What are special considerations when using a behavioural marker system for assessment?
The use of a behavioural marker system in a formal assessment of non-technical aspects of performance presents significant challenges. The behavioural marker system must capture the context in which the assessment is made (e.g. crew dynamics and experience, operating environment, operational complexity). For example, in a team endeavour, the behaviour of one crew member can be adversely or positively impacted by another, resulting in a substandard or inflated performance rating. Behavioural marker systems should be designed to detect and record such effects.
10 What are prerequisites to be a trainer for a behavioural marker course?
Qualifications required of the persons who will deliver a formal course to train, calibrate, and qualify raters (evaluators) using the behavioural marker system:
• Commitment to human factor’s principles;
• Domain knowledge;
• Formal training in applicable aspects of human factors or non-technical skills (e.g. Crew Resource Management);
• Formal training in the use and limitations of performance rating systems;
• Formal training in the use of the specific behavioural marker system.
11 What are prerequisites for evaluators using a behavioural marker system?
Entry requirements for personnel who will serve as evaluators:
• Commitment to human factors’ principles;
• Domain knowledge;
• Formal training in applicable aspects of human factors or non-technical skills (e.g. Crew Resource Management).
12 What are necessary qualifications of evaluators?
• Complete initial training in behavioural marker systems;
• Formal assessment as competent and calibrated following behavioural marker system-training in classroom;
• Calibration in operational environment (e.g. training, simulator, work environment);
• Periodic re-calibration for continuing use of the behavioural marker system.
13 What should the content of behavioural marker system training be?
• Make explicit goals for use of the behavioural marker system;
• Explain the design of the behavioural marker system, as well as content and guidelines for its use;
• Review main sources of rater biases (e.g. hindsight, halo, recency, primacy) with techniques to be used for minimisation;
• Present the concept of inter-rater reliability and the methods to be used to maximise it;
• Illustrate and define each point of the rating scale and different levels of situational complexity with video examples, discussions, and hands-on exercises;
• Provide practical training with multiple examples;
• Include calibration with iterative feedback on inter-rater reliability score;
• Teach debriefing skills as appropriate;
• Conclude with a formal assessment of rater competence.
14 How should a behavioural marker system training be structured?
• Minimum two consecutive days training;1
• An ideal group size of 8-12 people;
• Training follow-up (e.g. meetings, feedback via telephone) after use of behavioural marker system in operational setting;
• Training ideally utilises video examples from the organisation.
15 What training and calibration materials should be used?
• Videotapes of scenarios with professional sound and visual quality;
• Demonstrating various levels of performance;
• Showing all behavioural markers in scenarios illustrating various environmental conditions and complexity;
• Depicting increasing lengths of segments with training progress (e.g.
from 2 minute vignettes of a specific behavioural marker to an entire flight/surgical procedure).
• Information about the background of the behavioural marker system with full reference documentation.
16 What are regulatory issues regarding the use of behavioural marker systems?
• The rationale for employing a behavioural marker system in any domain is to improve levels of safety and to facilitate attainment of the highest possible levels of performance;
1. Note: This is a recommendation from the GIHRE report and not a CAP 737 recommendation.
• A partnership between the operators and regulatory authorities is needed to achieve equitable assessment of non-technical skills, especially when a pass/fail criterion is mandated;
• Regulators should move cautiously when initiating formal assessment of non-technical skills.
17 What are research issues regarding the use of behavioural marker systems?
By nature, behavioural marker systems are not static, but must be continually evolved or refined in response to changing operational circumstances (e.g.
development of equipment) and increased understanding of human factors issues in the domain. The following list, which is not exhaustive, specifies research topics where empirical evidence is either lacking or incomplete and systematic research should prove highly beneficial:
• Developing empirical evidence for the relative merits of global vs. phase or event-specific ratings and individual vs. team ratings;
• Defining context effects on crew behaviour and developing a systematic system of integrating these measures with behavioural markers to provide a more comprehensive system;
• Investigating the distribution of ratings of markers taken in different data collection environments (i.e. training including technical and non-jeopardy, full mission simulation (LOFT), non-jeopardy assessment of system performance (LOSA), formal evaluations in both line operations, and recurrent proficiency checks);
• Integrating knowledge from incident analyses, especially coping/recovery strategies and translating them into behavioural markers;
• Providing practical guidance for the transfer of behavioural marker systems and/or their components across domains and cultures (national, professional, and organisational).
Conclusion
• Behavioural marker systems have demonstrated value for training, understanding of performance in high risk environments, and research into safety and human factors.
• Behavioural marker systems can contribute to safety and quality in other work environments, as well as in high risk settings.
• Concepts are continuously evolving as a result of co-operation between practitioners and researchers.
• Researchers, practitioners, and regulatory authorities must work congruently in order to realise the ultimate goal of improved safety.