Training Sessions
Class size and average driving times per training session
Total Class Size Average Drive Time (in hours) per Successful Trainee
Table 2: NBOT training sessions highlights
Average drive time with an Instructor (in hours) 40:42 26:34 44:31
Total NBOT Graduates who are still Active5 96 55 75
# (%) of active NBOT graduates who had a
preventable collision since hiring (includes training
time) 20 (21%) 22 (40%) 15 (19%)
# (%) of active NBOT graduates who had a
preventable collision during training 3 (3%) 9 (16%) 1 (1%)
# (%) of active NBOT graduates who had a
preventable collision during training who were later
involved in a repeat preventable collision 2 (67%) 6 (67%) 0 (0%)
# (%) of active NBOT graduates in a preventable
collision within 4 months of graduation 21 (22%) 17 (31%) 21 (28%)
# of preventable collisions within 4 months of
graduation per active NBOT graduate 0.27 0.35 0.35
Some of this difference might be explained by the time of year when NBOT sessions 12-17 to 17-17 graduated. Those sessions graduated from October 31, 2017 to
December 27, 2017. As such the graduates would have spent some of their initial four months driving in winter conditions. However, when we compared their accident rates to those that graduated at a similar time in 2018, the rates for the 2017 graduates are again higher. The percentage of active NBOT graduates in a preventable collision within 4 months of graduation was 31% for the 12-17 to 17-17 training sessions, versus 27%
in 2018. This is consistent with the observation above that the drivers trained in sessions 12-17 to 17-17 had relatively more preventable accidents.
4 Does not include Platform Mentor time. Training and Development were unable to provide these times for certain session in 2017.
5 The above analysis includes only operators who were still active at the time of our review in order to reduce the impact on the data caused by different employment periods (e.g. temporary or short-term employment).
Ongoing training deferred or not scheduled
We found that the influence of Operations, including its need to adjust due to the delays in opening O-Train Line 1, resulted in ongoing training being deferred or not scheduled at all. The details of training that was deferred/not scheduled are discussed below in more detail under Objective #2.
Operator Certification and Training Plan
To augment the governance over training of operators and as part of OC Transpo’s commitment to improvement, a new guiding document, Operator Certification and Training Plan (Certification Plan), was issued in September 2019, after we started this review. The Certification Plan outlines the certifications for all training programs, both bus and rail and has been approved by the Chief Safety Officer, Director Transit Operations (OC Transpo and Para Transpo) and the Director Rail Operations. The Certification Plan is an important step in defining governance over training and
recognizes the importance of committing to a process of providing training to operators.
We believe that there is room to improve the Certification Plan by including:
• Authorities and process to adjusting course content and schedule in order to meet operational requirements;
• Provision that Operations make relief instructors available as per training schedule;
• Linkages between Training and Development and Operations in the development of training schedules;
• Agreement that drivers will be made available for training as scheduled by Training and Development; and
• Apart from the Defensive Driver Course modules – the Plan does not provide additional information on the frequency and scheduling of other mandatory and non-mandatory training for ongoing bus operators.
In our view, sound governance over bus operator training reduces the risk of preventable collisions by bus operators.
Recommendation #5
That the City continue with recent efforts to augment governance over bus operator training that training, as a safety critical function, is implemented as designed and without interference.
Management response:
Transit Operations and Training work together to balance OC Transpo’s service requirements and safety-critical training.
Management agrees with this recommendation and beginning in May 2020, staff from Transit Operations and Training increased the frequency of their meetings to weekly, to plan ahead and ensure that both OC Transpo service and bus operator training
requirements are met.
The Operator Certification and Training Plan mentioned in this report is being updated to clarify governance over training, strengthen collaboration between Operations and Training, and include the frequency and schedules for all mandatory and
non-mandatory training. Revisions will be complete by Q4 2020.
Driver Transition to Operations
OC Transpo strives to ensure that upon graduation, operators passing NBOT have been trained to operate any type of bus, on any route, at any time. As NBOT progresses the difficulty in routes driven progresses; trainees begin in low-traffic suburbs and then move on to more high-traffic areas, like the downtown core. Training and Development provided us with a list generated by Fleet Safety of the 12 routes with the most
preventable collisions.
We expected to find the experience of a new bus operator to be a key determinant in the type of bus and the difficulty of the routes that an operator will drive once they have graduated from NBOT as not all routes are equally difficult. What we found was that as per the City’s negotiated collective agreement, the routes and type of bus booked by all operators are determined by their seniority within their organization and not by their experience as an operator or the level of training received by the bus operator. As a result, the skills of the bus operator do not factor into the booking of their work and new bus operators can select to operate more difficult routes.
In our view, by not aligning the work that is booked by new operators to the experience level of the bus operator, the risk of preventable collisions by bus operators is
increased. However, we recognize that due to the collective agreement with ATU 2796, bus operators book routes based on seniority, meaning that management could not
6 New agreement with ATU 279 expires in 2025
implement such a change if it wanted to. We also considered examining possible changes to higher risk bus routes to reduce the risk of accidents, however this was outside of the scope of this review.
Training standards NBOT
OC Transpo has a robust training curriculum focused on safe driving practices. The program includes a combination of in-class and on-road driving time. During new bus operator training, the program is designed such that trainees operate all bus types in the fleet (i.e. 40-foot, articulated and doubler-decker). Before trainees can take the MTO on-road driver exam, they must successfully complete at least 20 hours of driving under the supervision of an instructor who has delegated signing authority from MTO. Additionally, prior to receiving their C class license trainees:
• Complete certain elements of defensive driving training, written tests and daily practical evaluations;
• Receive training regarding their obligations to follow all the rules of the road; and
• Pass a formal MTO road test, consisting of a pre-trip inspection, backing procedure and driving on a prescribed test route approved by MTO.
The MTO portion of NBOT ends at approximately 60% (Day 15 or 16) into the NBOT session. The remaining NBOT days focus on adding to skills already learned and on building the skills necessary to be a successful OC Transpo bus operator.
The review expected to find the NBOT program clearly defined, updated and effectively addressing training requirements. We found that the NBOT curriculum was clearly defined through a series of curriculum documents (95 binders) and a supporting syllabus for each training session. The complete NBOT curriculum covering both the MTO and the subsequent OC Transpo specific training has been approved by MTO.
The most recent MTO audit of the delivery of the DCP portion of NBOT was in 2017 and no infractions were noted.
Training and Development staff track updates required to the curriculum/syllabus and they inform MTO of major changes to the curriculum/syllabus. Full-time instructors are required to undergo significant training and re-training.
As part of effectively addressing training requirements, the review expected to see objective, defined training and performance standards and criteria that are adhered to in order to determine successful versus unsuccessful NBOT trainees. We found that while
MTO standards are defined, “OC Transpo” standards which are often referred to by Training and Development are not formally defined and documented.
We believe that there is an impact to not having clearly defined, objective, training standards. The lack of such standards increases the risk of certifying trainees as new bus operators and permitting them to operate a bus in-service when the trainee is not yet ready to operate a bus in-service.
Pass/Fail decisions
We found that, based on information provided by Training and Development, very few NBOT terminations of trainees were due to trainee’s failure to meet OC Transpo
driving/safety standards. The majority of performance-based terminations were due to a failure to meet MTO standards. Specifically, 85% of terminations from 2015-19 were clearly due to a failure to meet MTO standards and only 8% of terminations from 2015-19 were clearly due to failure to meet OC Transpo driving/safety standards7.
It is important to note that we expected the number of failures would decrease as new bus operators progressed through their training. This is because many individuals who obviously are not suited to be a bus driver can be identified relatively early in the
training process. In other words, to some extent the large share of terminations coming from failure to meet MTO standards is as we expected.
Training and Development staff monitor driving performance of trainees daily. However, other than the MTO exams, there is no concept of an “auto fail” nor specific criteria to determine a pass/fail and no formally defined performance standard that must be met.
Instead, trainee performance is assessed on a case-by-case basis. As a result, there is a great degree of judgement that is applied and this introduces the opportunity for risk of bias (perceived or otherwise), and inconsistency in evaluating trainees.
We reviewed a sample of 12 trainees’ files. In our review we found instances where comments, for example, were made about unsafe driving practices, violations under the Highway Traffic Act, failure to improve as they progressed through NBOT and an act that required the instructor to intervene. We would expect that this trainee would have been terminated from the program at this point, however, the trainee continued with, and in some cases, completed the program.
7 The remaining 7% were terminated for other reasons.
We found that the Daily Progress Reports (DPRs), while monitoring progress of trainees, does not define a standard or provide a direct relationship to decisions to pass/fail the trainee. This is in contrast, for example, with the Toronto Transit
Commission (TTC), where if a trainee in new bus operator training has two bad daily driving reports they are discontinued from the program.
Training and Development staff informed us that the process of removing a trainee can be lengthy and time consuming. Based on their past experiences, they need to be especially cautious to mitigate the risk of grievances or legal action. It can be easier to let the trainee fail the MTO exam twice and automatically remove them from NBOT for that reason, rather than go through the resource-intensive internal process of failing them for reasons outside of MTO requirements.
Training and Development staff informed us that they have directed instructors to clearly document issues on trainees’ DPRs so that there is evidence in case it is ever required.
We found however that DPRs are not always well documented.
We also found that preventable collisions occur at various stages during training and approximately 60% of these collisions occurred within 10 days of graduation. While we recognize that 77% of preventable collisions are deemed to be minor, in our view there is no clear standard or direction as to when a collision during training should trigger a termination or review.
Recommendation #6
That the City clearly define, document and apply objective “OC Transpo” criteria and standards for determining successful/unsuccessful NBOT trainees.
Management response:
While not required by our governing body (MTO), management agrees with this recommendation.
Management is reviewing the NBOT curriculum that takes place after the MTO road test and establishing clear, objective pass/fail criteria for instructors to use during this portion of the program. A new standard operating procedure that establishes the clear,
objective pass/fail criteria for OC Transpo training is in development. The review and supporting procedure will be complete by Q4 2020.
Daily Progress Reports
The Daily Progress Report (DPR) has been relied upon, for a number of years, as a very important tool used by Training and Development to rate and monitor trainee progress through each day of the NBOT. The DPR is a two-page form that should be completed by the instructor while the trainees are driving the bus. At the time of this review, the DPR rated the trainees on a series of skills that were to be observed while driving. The rating scale used was1 to 4 as follows:
1 – Skill was performed improperly/unsafely
2 – Skill was performed safely but requires improvement 3 – Skill was performed safely, smoothly and without errors
4 – Skill was not coached but requires more practice/Not-applicable
The actual rating of skills was a 1, 2 or 3 with the rating 4 relating to a skill which was not observed as being performed. Instructors were to complete the comments on the back of the DPR and all DPRs were to be signed by the instructor and the trainee. Other information appearing on the DPR included the type of bus driven, start time/end times, mileage in and mileage out, road conditions and total driving time.
We reviewed the DPRs for a sample of 12 trainees. We found that a DPR was
completed for all trainees daily. However, we found the DPR tool to be inadequate to track and evaluate trainee progress. Specifically:
• Training and Development informed us that instructors were required to state clearly on the DPR that all areas of the syllabus for that day were covered. We found that this was often absent on the DPR. If parts of the syllabus were not covered in the drive time, there was a section on page 2 of the DPR to note the absences. However, we observed in our testing that this section was, at times, not completed. Likewise, the DPR was to document a follow-up plan when areas of the syllabus were not covered. We found that this was not being documented.
• The DPR rated trainees on a scale of 1 to 3. We found that a scale of 1 to 3 does not provide the level of sensitivity to truly separate trainee levels of performance.
The vast majority (approximately 75%) of scores on the DPRs were a 2. Our analysis indicates that the scores of NBOT trainees who fail or end up having preventable collisions after NBOT completion were indistinguishable from those who passed NBOT and have no collisions, as depicted by the following chart:
Figure 2: Mean score by trainee outcome
• We did not find a relationship between a trainee’s average DPR score and his or her likelihood of passing NBOT. We expected to find average DPR scores
increasing or levelling off at higher levels the further a trainee progressed through NBOT as this would indicate that driving skills were improving. We did not see such a relationship. We found that in some cases trainees who successfully
completed NBOT were scoring, on average, less than 2 and at times averaging as low as 1.70 in the days just prior to graduation.
• There were no criteria documented to guide the instructors on how to assess the trainees progress against the DPRs rating scale of 1 to 3. This resulted in
inconsistent ratings provided by instructors on DPRs from day-to-day and from trainee-to-trainee.
• All skills on the DPR scoring sheet had equal implied weight, even though some are clearly more safety-critical than others. For example, use of the radio (e.g.
awareness, volume, protocol, talk group, communication abilities) carried the same weight as lane changes and left/right turns.
• DPR scores did not input into any broader data analysis to determine potential predictive results or improve the program.
• The grading boxes on each DPR were nearly identical and not tailored to the syllabus material for that day.
Pass + no accidents