Work Samples

The principle of a work sample is to create a task typical of the job to be performed, then observe the results and provide feedback. Work samples are typically used when screening applicants in the airplane (military) or when testing licensed pilots in a flight simulator. The value of work samples is highly dependent on the standardization of the exercises and the quality of observation personnel. If performed by well-trained and experienced experts, work samples can be of good value because of their realistic content.

7.2.8 SIMULATION-BASED TESTING OF OPERATIONAL COMPETENCIES

Simulation-based testing of operational competencies can combine realism of work samples with the advantages of psychometric testing. This form of testing addresses the ability of pilots to solve complex tasks in dynamic environments. Such scenarios are more realistic and comprise interactions of multiple requirements, which must be controlled by pursuing certain (professional) strategies. They require not only rational performance (strategies), but also include emotional (fears and fun) and motivational (confidence and commitment) aspects. Testing scenarios are quite complex and tend to be less rigid then psychometric testing. They also can capture the ability of problem-solving in unstructured situations (unexpected emergency situations which cannot be drilled by procedures). Simulation-based testing of operational competencies can be performed on specifically programed (PC-based) low fidelity simulators. They provide

5

Guidance Material and Best Practices for Pilot Aptitude Testing

7.2.10 FULL-FLIGHT SIMULATORS

Assessments in Full-Flight Simulators (FFS) seem, at first sight, to be ideal because they offer the highest degree of realism, by reproducing the actual dynamics and complexity of the pilots working environment.

Flight Checks in full-flight simulators are commonly used to test flying skills of ready entry pilots, first officers and captains and are valuable tools to complement, but not to replace aptitude testing.

Note: From a diagnostic point of view, work samples in full-flight simulators are quite demanding (standardization of the scenarios, disturbances, quality of observation, complexity of instruction, inter-rater reliability, etc.). Simple arrangements do not necessarily produce the kind and quality of data, required for valid aptitude testing purposes and are therefore not advised as replacements for classic means of aptitude testing.

7.3 MOTIVATION OF APPLICANTS

For reasons previously outlined, candidate motivation on entry is probably more important today than previously. Besides satisfying content and methodology, criteria tests should also be easy to use. Active pilots should be motivated to show their full potential during the tests. This can be achieved by constructing tasks which are perceived as being the following:

Ê Relevant to the job

Ê Low degree of difficulty to minimize disappointment Ê Fair with respect to the framing conditions

Instruction must be understandable and should challenge performance adequate to the measuring dimensions. Some motivational screening tests with strong success rates are available today. By design, the best of these ensure that rehearsed answers will not be successful.

7.4 IATA MATRIX – PILOT APTITUDE TESTING

A complete test battery consists of at least the following components, which are arranged in the given order:

Ê Screening (formal requirements) Ê Tests of basic mental abilities Ê Tests of operational competencies Ê Tests of social competencies

Ê Half-standardized interviews to capture relevant personality traits Matrix 1

Matrix 1 proposes one possible method to allocate measuring dimensions and instruments to the four target groups – other solutions are possible.

Measuring

Dimensions Instruments Ab-initio Ready Entry First Officer Captain

Screening

Basic mental Psychometric Memory capacity Logic abilities

Measuring Dimensions / Testing Instruments

Measuring

Dimensions Instruments Ab-initio Ready Entry First Officer Captain abilities Paper-pencil

tests PC-based psychometric tests

Speed and accuracy of information processing (perception,

Stress – coping with social confrontation Stress – coping with information load Stress – coping with time pressure Self-discipline

Self-criticism

Captains additionally: Safety motivation

Guidance Material and Best Practices for Pilot Aptitude Testing

Matrix 2

Matrix 2 shows in addition, a sample set of appropriate measuring scales.

Phases Measuring Dimensions Target Group Instrument Scales

Screening

Entry School grades Rank Rows

Test of basic mental

abilities See matrix above

Ab-initio, Ready

competencies See matrix above Ab-initio, Ready Entry, FO

competencies See matrix above All

Group tests by means

Test personality traits See matrix above All Half-standardized interview in connection with rating system

Rank rows, qualitative discrimination

8 DESIGNING PILOT APTITUDE TESTING SYSTEMS

Defining organizational need:

Designing an effective pilot aptitude testing system is a complex task with numerous considerations. It is not possible to cover all aspects in their entirety within this manual. The intention of the content is to examine several crucial aspects of the process. The most important starting point is to define the actual needs of your organization. Defining your needs determines what the selection system used shall deliver, called the criterion. Some examples could be the reduction of failure rates during type rating training, an improvement of service mindedness, cost reductions, less staff turnover, assurance of a certain level of quality, etc. If there is no identified issue(s), the organization may not need to install an additional system.

Sometimes simple aptitude testing procedures developed at the early development of the organization (sometimes called casual selection systems) have become recognized for their ineffectiveness, and the need for upgrading is understood.

Job analysis:

Job analysis involves two steps. First, the purpose of the job needs to be established (why the job is needed in the market and what function it serves). This step is called “Job description”. It must not be confused with the second step, which clarifies what kinds of abilities the applicants should possess to be able to perform the job well. These are called “personal requirements” and sometimes also called “job requirements”. Job analysis with job description and personal requirements are important because they require a detailed analysis of the organization’s objectives, company values and future challenges. This task requires significant management input.

It is important to establish a scientifically detailed definition of the essential employee qualities necessary for excellence in job performance. Besides basic general abilities, professional abilities and certain personality traits (sometimes operator specific) will be preferred.

System goals:

Once these prerequisites are established the organization can move toward building and maintaining an efficient and effective aptitude testing system capable of achieving the following goals:

Ê Identifying the most suitable staff for the job

Ê Delivering selected personnel at lowest possible cost Ê Providing a fair and legally defensible architecture

Guidance Material and Best Practices for Pilot Aptitude Testing