Verify instructional need

mission/job and the manner in which the mission/job is trained. After determining that there is an instructional need, determine whether the instructional needs are caused by the curriculum design, the student population, the instructors' qualifications, a new instructional need, or another instructional related cause. The following may help in verifying an instructional need:

a. Identify the extent and gravity of the problem. Obtain information from field commanders about the extent (widespread among a number of units or isolated) seriousness, safety, or security of problem and its impact on mission capabilities. b. Identify the impact on individual and unit performance. If there are no mission

consequences, there may be no need to pursue the matter further.

c. Assess reliability of the information. See Table 6 for factors to consider when assessing both formal and informal performance reports.

d. Determine statistical validity of the training requirement to verify whether events are chance occurrences.

TABLE 6. Assess reliability of performance data.

FOR THIS TYPE OF INFORMATION

CONSIDER THESE FACTORS WHEN ASSESSING RELIABILITY

Formal Report Examples:

• Training effectiveness evaluations.

• Lesson learned reports.

• Availability or lack of supporting data.

• Sample size and use of statistical procedures.

• Degree to which assumptions can be made about performance throughout the Services.

• Unity of view vs. differing opinions.

• Degree to which report confirms other assessments and actions (e.g., known defects in current training courses, etc.).

• Precision. Is report based on task descriptions, or is it impressionistic?

• Logic. Do conclusions represent reasonable assessments of the performance data?

• Report experts. Were they the right experts? As a rule, look for an expert who is one rank and/or skill level higher than that of the performers.

Informal Information Examples: • Phone conversations. • Discussions with performers at school or first-time performers. • Discussions with performers in the field.

• After action reviews.

• Personal knowledge of the personnel providing the information.

• Level of detail and relationship to task documentation. The more explicit the information, the more likely it is to be valid.

• Availability or lack of support data.

• Sample size and degree to which assumptions can be made about performance throughout the Services.

• Unity of view vs. differing opinions.

• Degree to which report confirms your hypothesis.

• Degree to which report confirms other assessments and actions (e.g., known defects in current training courses, etc.). e. Identify the major causes of the performance deficiency. Causes for unacceptable

performance are usually deficiencies in doctrine, training, leadership, organizational, and/or materiel (equipment) deficiencies. Table 7 lists indicators of problem causes that may be related to the performance deficiency. Review your evidence in light of these indicators.

TABLE 7. Indicators of problem causes.

INDICATOR CAUSE POSSIBLE

SOLUTION • Practice is infrequent or unrealistic.

• Performance is complex and has no job aid.

• Task/equipment is new.

• Individuals do not possess required knowledge to perform task.

Training deficiency • Training. • Information. • Job aid. • Coaching. • Mentoring.

• Individuals not in appropriate job for their training.

• Individuals not getting clear/timely feedback on performance.

• Lack of clarity on role/job in unit.

• Reward system is inadequate.

• Tasks are distasteful.

Leadership deficiency

• Information.

• Leader training in coaching, mentoring.

• Work facilities inadequate.

• Barriers to performance present.

• Personnel shortages.

• Supply and demand difficulties.

• Organizational makeup not adequate for mission requirements.

• Frequent leader changes.

Organizational deficiency • Refer appropriate information to the unit chain of command.

• Individuals perform as expected but mission still not achieved.

• Equipment constantly broken.

Materiel deficiency

• Refer appropriate information to cognizant personnel. f. Training can only solve knowledge and skill deficiencies. Problems relating to

environmental or motivational factors indicate a performance deficiency in one or more of the other causal areas and cannot be solved simply by training.

g. Review any previous test results and course evaluations. Do the LOs, as stated, reflect the actual job performance requirements? Can the individuals perform the objectives to the stated standard at the end of training? Look for information suggesting that the whole course may have some problems, as opposed to a particular session or instructor or student(s) having a problem. See Table 8 for suggestions on how to evaluate certain common types of evaluation information. Check for information indicating that tests have been validated (i.e., that students who pass tests can perform effectively and those who do not pass cannot perform). If no data exists, focus the rest of the course review on ways to make the course more efficient and more reflective of anticipated individual performance in the future.

TABLE 8. Reviewing evaluation information about school courses. TYPE OF EVALUATION INFORMATION CONSIDERATIONS Student attrition rates

• Identify trends. Attrition rates staying about the same? Improving? Worsening?

• Identify causes. Where rates are high, administrative reasons may cause the dropout rate such as failing the physical fitness test. Special training

times

• Ensure the student evaluation plan includes remediation details.

• Identify remedial training in current and previous courses and its extent. Such remediation may be taking place without overall course length being extended (for example in the evenings).

• Review the total training day.

• Review the plan of instruction for massed training right before a test (could indicate insufficient previous training or poor training design). High failure rates

for certain tests

• Review test data.

• Do not change the standard unless it is clearly inappropriate; look for the possibility that certain tests are causing problems. Poorly written tests may cause failures. If the test is valid, then check the training design and presentation.

Student critique sheets

• Identify trends between classes and over a period of time.

• Keep the perspective of the students in mind. For many courses, they may lack experience to reach a balanced judgment.

• For leadership courses, critiques will often reflect experience.

• Be alert for a grouping of complaints around particular tests, instructors, or training sessions.

• Pay attention to students saying they "still do not understand." Even if they do in fact understand, such expressions indicate a lack of confidence, which in itself may cripple performance.

h. Identify how well individuals should be performing at the end of training and on the job. Task analysis and course design documents provide the original intent of the course (as stated in objectives). Evaluate that intent against current individual performance at the end of training and in the field. Failure to perform this step often leads to making changes that do more harm than good. Consider holding a review with a team of SMEs, training developers, and instructional system specialists. As a rule, select individuals who have recent field experience and who are at least one grade and skill level above the grade/skill level being taught in the course.

i. Identify strengths and weaknesses of course tests, including hands-on tests. Vital factors to look for include:

(1) Terminal LOs that are tested. (2) Use of hands-on performance tests. (3) Use of criterion-referenced tests.

(4) Written tests, where used, are valid (see Table 9). (5) Whether tests realistically measure performance of LOs. (6) Tests allow use of job aids when they are used on the job.

(7) Decision to use trainers in tests is for the sake of costs, safety, or enhanced realism. (8) Test items with high failure rates are reviewed.

TABLE 9. Using written tests.

QUESTIONS ABOUT WRITTEN TESTS

CONSIDERATIONS

Is the performance that is being measured paper and pencil performance?

When the performance consists of filling in forms, correcting memos, preparing letters for commanding officer signature and the like; paper tests are the obvious choice.

Is the test being used to indirectly measure "thinking" behavior?

Often, thinking behavior is best measured using paper tests. For example, problems, which are primarily mathematical in nature often, are easily and accurately evaluated with paper tests. Examples include: ammunition expended at different rates of fire, distances that can be traveled given current fuel levels, etc. Also, memorized performance can be measured using paper (e.g., specifying from recall the five steps of medical assistance on the battlefield, etc.). High level knowledge test examples. Cognitive process, problem solving, mission planning. Is written testing being

substituted as a less costly alternative to other forms of testing?

Often, costs drive the decision to use paper tests. When this is the case, look to see if the performance is tested elsewhere in the course using more realistic evaluation devices. Or, look to see if acceptable performance of a task can be inferred from the hands- on tests of other tasks. For example, in leadership courses, case studies are often used to evaluate an individual’s ability to respond to a variety of different situations, combat or otherwise. Does the written test

measure anything useful?

Multiple choice tests can be used for measuring the knowledge of a student prior to entering a course, reinforcement of content knowledge, placement, post-testing etc. As a rule, multiple choice tests can accurately measure "recognition" levels of performance while using them to measure "recall" and "hands- on" performance always involves an inferential leap.

6.2.5.2 Assess non-instructional issues. Assess the effect of non-instructional issues on the