LEVEL 3—ON-THE-JOB PERFORMANCE

Since reproductive health training courses usually focus on developing or improving clinical skills, the trainer will want to be assured that, after the course, the participant is using the skills acquired. Followup evaluation reveals whether or not changes in the individual’s behavior and performance on the job have occurred subsequent to training. While the instructional designer may not be involved in preparing for or conducting a Level 3 evaluation, the designer should be familiar with this type of evaluation.

If change in provider practices following training is to be documented, comparable information at the facility and/or the individual provider level should be collected as a baseline before the course begins. One way of accomplishing this would be to collect performance data during a needs assessment visit before the course begins.

The purpose and content of followup evaluations should always be discussed with participants before the course ends. Knowing that a followup evaluation is planned may motivate participants to use the skills they have learned.

Participants’ performance after training can be evaluated by observing them on the job, by interviewing them and by interviewing their supervisors. Followup evaluations typically occur after the individuals have had sufficient opportunity to apply their new knowledge and skills on the job (e.g., 6 to 12 months after training). Following are some basic questions which should be asked when evaluating how effective training has been in improving an individual’s job performance.

! Is the individual able to perform competently the skills s/he was trained to perform?

! Has the individual been given an opportunity to practice the new skills?

! Do supervisors, managers or those who received training feel that the course made a difference in the quality of their work?

! Did training solve any problems or fill service provision needs identified during the needs assessment process?

By answering these and similar questions, the trainer and the organization sponsoring the training can:

! Determine to what extent the desired training outcome has been achieved

! Identify discrepancies that still exist in job performance ! Plan for additional training if needed

Site Visit Observations

Observing individuals through site visits to their facilities is an important way to determine whether or not they are using the skills learned during the course. The individual and supervisor should be notified that a site visit will be conducted and told the purpose of the visit, when the visit will occur, who the observers will be and what preparation is needed (e.g., scheduling clients for skills demonstration).

The data collection tool for the site visit needs to target the skills the individual was taught during training (i.e., those listed on the competency-based checklists used in the course as described in Chapter

6). Sample 9-6 is an example of a followup site visit form that can be

used to assess the family planning skills of service providers. When assessing clinical skills, the trainer should take an anatomic model in case no clients are available.

By reviewing data collected through observations, the trainer can identify factors that influence the effectiveness of training. Recommendations for modifying the training course or participant selection will then become evident. If specific job deficiencies are identified post-training, they should be compared with data collected from initial assessments to determine the possible reason(s) for the current deficiencies.

Interviews with Those Who Completed Training

Health workers who completed training can provide data on the usefulness of the course to their current job functions as well as their ability to apply acquired skills after training.

The trainer may elect to conduct a structured or an informal interview or both to collect these data. Both data collection methods should focus on the skills emphasized during training and any constraints the individual has faced in applying these skills post-training. Sample 9-7 is a short questionnaire that can be used to document information about a facility’s family planning activities in general, as well as information about the individual’s family planning activities within that facility. General

information regarding activities conducted within the facility is useful because it provides a context in which to assess the individual’s activities. Data from this type of form provide some indication of:

! Whether or not the individual is applying the skills acquired during the course

! Constraints on the extent to which skills acquired during the course can be applied post-training, and their possible solutions

! The potential for training to have a positive effect on reproductive health services delivered at individual institutions

Interviews with the Supervisor

In most instances, the supervisor plays an important role in identifying the need for training and arranging for an individual to attend a training course. The supervisor is thus usually in an excellent position to provide feedback regarding the effectiveness of the course.

The trainer can conduct informal or structured interviews with supervisors during post-training followup site visits. The following are some interview questions the trainer might ask the supervisor:

! Is the individual able to perform competently the skills learned during training?

! How has training affected the individual’s attitude towards her/his work?

! From your perspective, what were the individual’s feelings about the training course?

! Were the changes in the individual’s performance worth the time spent away for training?

! What suggestions, if any, has the individual made to you about modifying future courses of this type?

! Did training correct the problem or meet the need for which the course was designed?

A review of the site visit responses of those who attended the course and their supervisors will identify ways of modifying and improving future courses. If most of the feedback is positive, it can be assumed that from the participants’ and supervisors’ points of view, the training has been effective. If specific concerns are expressed, compare these concerns with

data collected from other evaluation sources and try to address them before additional courses are offered.