For paratransit, quality is defined as: “A variety of measures meant to denote the quality of service provided, generally in terms of total travel time or a specific component of total travel time.” (Urban Public Transportation Glossary 1989). As early as 1975, Tomazinas documented the importance of quality of service to users of transit services. He concluded that transit
customers consistently rate quality attributes higher than standard efficiency or effectiveness measures. He also noted that traditional methods of measurement of efficiency and effectiveness are often inversely related to quality measures. This can mean that a system seen as perfectly efficient from the transit provider’s point of view is completely inefficient from the users point of view.
As noted in Section 4, much of today’s foundation in performance measurement was presented in Tomazinas’ (1975) work which drew together nearly all of the measures still used today.
Tomazinas argued for 14 quality measures, or in his terminology, efficiencies from “the point of view of the user”. These are presented in Table 21, which is abbreviated to exclude three purely fixed route measures.
Table 21 Quality Measures from Tomazinas (1975)
Cost of Travel Comfort of Service
• Direct Cost of Conducting Transportation per Vehicle Mile Carried
• Total Travel Cost per Unit of Distance Traveled
• Floor Area of Vehicle per Passenger Carried
• Ratio of Average Number of Seats Available to the Average Number of Passengers Carried
Quality of Travel Reliability
• Proximity of Service
• Frequency of Service
• Number of On-Time Arrivals per all Movements
• Ratio of Actual to Demand Frequency of Departure
• Number of Major Delays in Performance per all Movements
Safety and Security
• Number of Miscellaneous Accidents per Vehicle Mile
• Number of Fatal Accidents per Vehicle Mile
Methodologies that assess quality necessarily reflect subjective standards and measurements.
The question has been posed: “Which standards should be used?” The answer has been addressed by studying the opinions of the “experts”, the users, service providers and through attempts to synthesize combinations of those responses. An analysis completed by Pagano and McKnight (1983) of the relationship between the range of quality measures contained in the three groups mentioned above documented a strong correlation between the views of those groups, with the strongest link between responses of users and providers of service.
The inescapable subjectivity of most quality measures has fostered arguments against the use of qualitative measures, although Tomazinas (1978), Pagano and McKnight (1983), and Casey (1994) acknowledge the value of subjective data, especially when linked to related quantitative measures. While on-time service is intuitively ranked as the most important attribute of quality service, surveys have identified several quality issues that cannot be derived from standard efficiency or effectiveness indicators. The list of these attributes includes: driver friendliness, feeling of safety, driver kindness, comfort, and courtesy of telephone operators (Dimension Research, Inc. 1998; Pagano and McKnight 1983).
Necessarily, most reporting is oriented toward revenue changes, and the majority of transit performance studies have focused on related efficiency and effectiveness measures. Stone (1994), Pagano and McKnight (1983), and Casey (1994) propose that those two dimensions alone are unable to fully describe agency performance, and endorse adding the subjective measure of quality to any realistic evaluation.
An effort undertaken by the Chicago Transit Authority (Mumayiz 1987) gives some comparative on-time data. An analysis of both objective data and customer attitude was completed in this effort. Table 22 displays customer perceptions of on-time performance in the study. “On-Time”
here is defined as plus or minus ten minutes from the promised time.
Table 22 Customer Perception of On-Time Performance
Perception Pick-up Drop-off
Always On-Time 41.5% 39%
Usually On-Time 47.6% 48.6%
Seldom On-Time 7.0% 8.6%
Never On-Time 3.8% 3.8%
(Mumayiz 1987)
Table 23 displays Carter-Goble and Assoc.'s (1982) quality measures. Service quality measures in this category are an attempt to quantify a qualitative aspect of transit performance. These kinds of measures may be dependent on the use of passenger surveys to collect useable data.
The authors note that quality data may be the least affected by external macroeconomic variables, since financial data is not involved.
The 'Level of Service' measures are some of the broadest indicators in the Carter-Goble and Assoc. (1982) study. The authors suggest that their best use is for goals comparison on an annual basis. The measure Revenue Miles ÷ Revenue Hours is equivalent to average speed and is a quality measure because it is a component of the calculation for time spent in the vehicle;
slower average speeds may adversely affect passenger attitudes. It should be noted, though, that increases in average speed are not always a positive outcome: fewer boardings and stops
(decreasing load factors) may indicate falling system productivity.
Under the 'Safety' heading, ‘avoidable vehicle accidents' mean accidents resulting from
infractions of either a motor vehicle law or a system policy by the transportation system's driver.
Safety measures are most often compared either to internal goals or to each other in a time-series analysis.
The 1982 Carter-Goble Associates report was submitted for use by rural and small urban transportation staff as a guide for evaluating their system's performance. The report is oriented toward fixed route service. In this study, Carter-Goble Associates (1982) suggest that "on-time"
means zero minutes early and no more than 5 minutes late at least 90% of the time.
Their research lead them to suggest that the ideal complaint level is no more than three per individual driver per year. Carter-Goble and Associates (1982) present one of the few attempts to measure availability of accessibility features and vehicle cleanliness. A distinctive variance between Miller (1989) and Carter-Goble Associates (1982) is in safety measures; Carter-Goble Associates (1982) suggest one accident per 18,000 vehicle miles as a goal - Miller (1989) suggests one per 1,000,000 miles.
Pagano and McKnight (1983) developed a quantitative method to measure quality of service to disabled riders. The authors collected data from users of paratransit services, experts in special transportation, and service providers to develop an exhaustive list of both the important quality attributes and their relative importance. The study presented an index of measures based on information gathered from the experts and user questionnaires, and then an input utilization (production function) of the proportion of resources used by providers. The study found that measures of quality from all three groups (users, experts and providers) were highly correlated.
Table 23 Quality Measures from Carter-Goble, Assoc. (1982)
• (G) Revenue Miles ÷ Revenue Hours (average speed)
• (G) Vehicle Miles per Year
• (G) Vehicle hours per Year Safety
• (G) Vehicle Miles ÷ Number of Revenue Vehicle Accidents
• (S) Non-Vehicle Accidents per Year
• (S) Avoidable Vehicle Accidents per Individual Driver
• (S) Revenue Vehicle Accidents for Each Route
• (S) Revenue Vehicle Accidents for Each Vehicle ("G" – General, "S" - Supplemental,)
The potential weakness of utilizing judgment in quality assessments was addressed using a framework that incorporates judgment into a methodical decision process. A methodology used to quantify subjective input was based on Miller’s (1978) work suggesting a five-step process.
The model, combined with the Pagano and McKnight (1983) study is abbreviated as follows (from Pagano and McKnight 1983):
1. Establish a list of criteria, which is complete, mutually exclusive, independent, and of major significance. Accomplished through:
• Literature Review
• Experience
• Expert opinions
2. Determine a hierarchical structure of successfully more specific criteria.
• Grouped into eight categories (Table 24)
3. Select physical performance measures for each lower-level criterion.
• Weighting to assign relative importance to both the category and individual attributes contained in the category.
• Psychometric scaling
4. Establish a scoring function or means of evaluating each alternative for each performance measure attached to a criterion.
• Designed to yield a score ranging from zero to one using a formula summing weights of categories (aspects) and attributes.
5. Validate weighting and scoring functions by applying them to real alternatives.
• Expert review and customer survey
• Apply to paratransit properties
By augmenting this framework with production function analysis and extensive personal
interviews, the Pagano and McKnight (1983) study identifies eight general categories or aspects of quality Table 24) as it relates to the paratransit industry. Each aspect has associated with it from four to eight attributes.
Table 24 Eight Aspects of Quality
• Reliability • Vehicle Access
• Comfort • Safety
• Convenience • Driver Characteristics
• Extent of Service • Responsiveness
A major portion of McKnight and Pagano’s (1984) and Pagano and McKnight ‘s (1983) studies were centered on the development and use of indexes of quality of service based on responses of paratransit users. The quality index was built on the eight aspects of paratransit service, shown in Table 24 and Table 25. Each aspect is listed in italics, followed by its associated attributes.
Each aspect weight is listed in the ‘Weight” column, also in italics, on the left side of the column in order of weight value. Attribute weights are listed along the right side of the column also in order of weight value. The weights are measures of the relative importance of the aspects and attributes of quality of service. The most important aspect is reliability and on-time performance with a 1.75 weight from the users point of view. Within that aspect, the attribute of arriving at destination on time had the highest weight. Attributes that are likely to be affected by computer assisted scheduling and dispatching are denoted by an asterisk. The authors used a survey of users and a psychometric scaling technique to assign the weights.
Ultimately, scoring functions were developed which measure the achievement of satisfaction for that attribute. Development of many scoring functions was a straightforward process; the measure of quality for instance for the existence of an open seat or appropriate wheelchair restraints is a simple “Yes” or “No”. Some more difficult to quantify measures are courtesy, cleanliness and friendliness of the driver.
The measures were then applied to forty-two paratransit operators in the Chicago area. An interesting outcome of the Pagano and McKnight study (1983) was that larger private paratransit providers scored higher on quality measures, while publicly managed properties score generally lower on the same measures. The project incorporated substantial analysis and development of several new measures of quality.
Table 25 Aspects and Attributes of Paratransit Quality (Pagano and McKnight 1983)
Weight Aspects and Attributes 1.75
Arriving at destination on time or within a few minutes of scheduled time Notification of delays or cancellations of service
Wait time (from time of reservation or schedule) for pick-up at home
Wait time (from time of reservation or schedule) for pick-up away from home Few delays while on vehicle
1.61
Safe driver/Low probability of traffic accident Low probability of falling
Low probability of personal assault The type of tie-down
The position of the wheelchair in the vehicle 1.36
Short distance from house or destination to vehicle Number of steps
Assistance in getting from vehicle to destination Assistance in carrying packages
The width of the isle
Presence of wheelchair lift or ramp 1.31
Knowledge of general needs of elderly and disabled users Courtesy an friendliness
Ability to handle medical emergencies Neatness and professionalism
Familiarity with habits and needs of individual user 1.29
Being picked up at times selected by traveler rather than at preset times Shortness of reservation time
Convenience of return reservation procedure Accommodation to changes in reservations 1.07
No or few restrictions on where vehicle will go
Total number of hours during which service is available
Low rate of turning down reservations because of limited capacity Service on weekends
Courtesy and friendliness of telephone operators Ease of getting clear information on service Receptiveness to complaints and user suggestions Procedure for following up on complaints 0.74
A guaranteed seat or location for wheelchairs The condition and cleanliness of the vehicle The smoothness of the ride
Air-conditioning and good ventilation
Sheltered waiting areas for pick-ups away from home Seats in waiting areas for pick-ups away from home (Adapted from Pagano and McKnight 1983 and McKnight and Pagano 1984)
(“*” denotes that attribute may be affected by Computer Assisted Scheduling and Dispatch)
A short discussion of selected individual attributes and their scoring functions is provided below.
These attributes were chosen on the basis of the likelihood of being affected by the introduction of a computer assisted scheduling and dispatching system (CASD). The scoring functions are grouped under their respective aspect of quality in the same way as in Table 25.
Reliability and On-Time Performance
1. Arriving at destination on time or within a few minutes of scheduled time – The authors of this study note the lack of accurate and/or complete records useful for evaluation of transit properties. Due to the lack of useable data, this study used the observations of the transit manager obtained through interview and researcher observation. Measures were developed using the following scale:
Measure of attainment Score
Good – 90% or over within 5 minutes 0.85 Fair – 80% to 90% within 5 minutes 0.65 Poor – Less than 80% within 5 minutes 0.35
2. Notification of delays or cancellations of service – Pagano and McKnight (1983) found that providers’ attention to this quality issue ranged from no notification (especially during inclement weather) and in the case were no reservation is taken, to those who make every effort to notify passengers in case of delay. The greater the effort, the higher the score, which is computed using the function:
Score = 1-0.02(x-5 minutes)
where: x = minutes from pick-up time or window that vehicle is expected to be late in order for the provider to notify.
Figure 1 graphically illustrates the scoring function for Notification of Delays or Cancellations.
3. Wait time (from time of reservation or schedule) for pick-up at home – Because no measure of actual wait time was available, the base for the score is the score for arriving on time. Each minute of window allowed for arrival time reduces the on-time score by 0.01. This means that if a provider is “Good” (90% or better on-on-time rates), and has a 15-minute window, then the score of 0.85 is reduced by 0.15. The complete score is: 0.85 – 0.15 = 0.70.
4. Few delays while on vehicle – The authors suggest that most delays occur either from pick-up and drop-off actions or from mechanical breakdowns. Scores for the two potential delays (delays due to passenger pick-up/drop-offs and breakdown delays) are calculated separately using the following scoring functions:
0.0
Minutes Provider Projects Vehicle Will be Late
Score
Figure 1: Scoring Function for Notification of Delays or Cancellations
(Pagano and McKnight 1983)
a. Expected delay due to passenger pick-ups/drop-offs: Calculated as the average occupancy minus 1, times the estimated time per pick-up. Estimated pick-up times are given for group pick-ups (30 seconds), door to door service (three minutes), lift passengers (one minute). This scoring function is:
S(P) = 1- 0.045 ((occupancy –1)(ETP + P(WC)) where: occupancy = average occupancy
S(P) = score for delays due to picking up other passengers P(WC) = percent of ridership using wheelchairs
ETP = estimated time per pick-up
b. Expected delay due to breakdowns: Based on the history of the provider.
Breakdown probability was estimated as the average number of in-service breakdowns per year divided by the total vehicle days per year. This function requires computation of both an estimated time per breakdown and probability of breakdown. The delay time was estimated with the following function:
where: ETb = estimated time per breakdown (in minutes) A = size of service area in square miles
R = 1 if vehicle is radio equipped, zero otherwise B = 1 if they have a back-up vehicle, zero otherwise
Expected delay: the probability of a breakdown times the estimated time per breakdown.
S(B) = 1- 0.5 (Pr(BD)) (ETb)
where: S(B) = score for delays due to breakdowns pr(BD) = probability of breakdown
Overall Score = .5 S(P) + .5 S(B)
5. Wait time (from time of reservation or schedule)for pick-up away from home – same as waiting at home.
Driver Characteristics
6. Familiarity with habits and needs of individual user – While suggesting that this attribute has value in assessing a paratransit agency, Pagano and McKnight (1983) where unable to accurately score this function because of the difficulty in measuring the driver’s attitude. They developed a rough scoring function using driver turnover rates as a way to estimate the relative familiarity a driver might have based on the number of times a driver comes in contact with a passenger through normal
scheduling. The authors developed three functions, one each for low, medium and high driver turnover rates. Because the ideal driver – passenger assignment is a permanent arrangement where the same driver transports the same passengers, those that do so are given a score of 1. When dispatchers cannot always assign passengers to the same driver, but try to do so, an extra 0.1 is added to the score. These scoring functions are shown below:
(Low Turnover Rate – 5 or more years of service per driver) Score = 1-0.05(D-1)
where D = number of drivers
(Medium Turnover Rate – 2 to 4 years of service per driver) Score = 1-0.1(D-1)
(High Turnover Rate – less than two years per driver) Score = 1-0.2(D-1)
Convenience of Making Reservations
7. Being picked up at times selected by traveler rather at preset times – A demand responsive service receives a score of 1.0, otherwise the score is computed using the following scoring function:
Score = 1-Pr(R)
where: Pr(R) = probability that passenger will not be able to take trip at preferred time. Pr(R) was
estimated as the average number of trip requests turned down or scheduled at another time of week divided by average weekly ridership.
8. Shortness of reservation time – Figure 2 shows the relationship between advance notice required for a reservation and the function “length of reservation”. Providers able to accept reservations in 30 minutes or less are given a score on “1”; those requiring seven days or more receive a zero. Because same day reservations are significantly better than a seven-day reservation requirement, a step function was used.
(Same Day)
Score = 1-0.075 (RH)
(One Day or More)
Score = 0.583-0.083(RD)
where RH = reservation time in hours where RD = reservation time in days
Figure 2: Scoring for Length of Reservation
(Pagano and McKnight 1983)
9. Convenience of return reservation procedure – This scoring function gives a score of 0.8 for an agency that accepts a reservation for both the pick-up and return at the same time and a score of 0.4 when customers are only able to make reservations for one way at a time. The authors note that the most convenient system is for no reservation to be required (e.g. fixed route); this receives a score of 1.0.
10. Accommodation to changes in reservations - This scoring function takes into account both the probability of being able to make a change and the shortness of the
reservation because being able to change the time or location becomes more
0.0
important as the length of reservation time increases. This interaction in represented by the function:
Score = 1 – (1-(P(A)) (1-S(R))
where: P(A) = probability of making changes S(R) = score for length of reservation
The development of the scoring function for accommodation to changes in reservations
necessitated the use of estimates derived from operator interviews for accommodating change.
The scale used to estimate the frequency for accommodating change, P(A) (based on operator interviews) was:
No problem in accommodating changes 1.0
Usually no problem 0.8
Will take emergencies 0.5
Try, but it’s difficult 0.3
Do not accommodate change 0.0
Scoring for Accommodation to Change is graphically displayed in Figure 3.
Figure 3: Scoring for Accommodation to Change
Extent of Service
11. Low probability of being turned down because of limited capacity – This score is based on the service’s turndown rate, so that zero turndowns are scored with a one.
The authors used their estimates of worst case turndown rates to develop the lower boundary at about 18%, or in the 10th percentile nationally. Using those criteria, the function is: Length of Time Required for Reservation
Score
No problem (1) Usually no problem (0.8) Emergencies (0.5) Difficult (0.3) Do not (0.0)
(Pagano and McKnight 1983)
Score = 1-5 TDR
where: TDR is the turndown rate (number of
potential passengers turned down per week ÷ ridership per week).
Responsiveness to User
12. Courtesy and friendliness of telephone operators – For this attribute, the authors assigned a subjective score based on their observations. An excellent score of 1.0 was assigned to an operator who is friendly and polite and who attempts to resolve all customer problems and complaints. An intermediate score of 0.5 is awarded if the operator is judged as “always polite … and does job.” The worst score, a zero, is
12. Courtesy and friendliness of telephone operators – For this attribute, the authors assigned a subjective score based on their observations. An excellent score of 1.0 was assigned to an operator who is friendly and polite and who attempts to resolve all customer problems and complaints. An intermediate score of 0.5 is awarded if the operator is judged as “always polite … and does job.” The worst score, a zero, is