Chapter 7 - Performance Evaluation
- No related sections.
To measure and communicate transit system milestones and their value is critical to convey the importance of transit to the general public, transit agency boards and the legislature. This plan uses two different types of metrics for measuring performance, those at the state level and metrics for local agency use. First, MnDOT uses performance measures to track progress at the state level, such a ridership. MnDOT also uses evaluation criteria to assess transit systems for strengths and weaknesses in order to make informed funding decisions.
In comparison, transit system guidelines and standards track progress at the local level and are controlled and monitored by the transit agency, separate from MnDOT. Transit system guidelines and standards recommended in this plan are the result of research into national peer systems, in addition to a survey of Greater Minnesota transit providers and cover operational metrics that help assess progress toward system goals and objectives.
The GMTIP’s performance evaluation strategy helps MnDOT and local transit providers to:
State Performance Measures and Evaluation Criteria
- No related sections.
Performance Measure: Assesses progress towards achieving a goal, outcome or objective. This definition covers performance measures used to make decisions or evaluate the effectiveness or adequacy of a policy, strategy or investment. Key transit system performance measures link MnDOT’s agency goals, the Olmstead Plan, Heading Home: Minnesota’s Homeless Initiative, and the GMTIP. The MnDOT performance measures also address the Federal Transit Administration regulatory mandates to develop measures and track performance.
Evaluation Criteria: Used by the Office of Transit to evaluate performance of transit providers. Evaluation criteria are used as criteria that helps inform MnDOT of system strengths and weaknesses. The evaluation criteria are also part of a larger evaluation for future expansion projects and capital needs
PERFORMANCE MEASURES
The GMTIP provides a series of performance measures and targets to guide MnDOT in response to federal and state mandated oversight requirements, plans, projects and investments.
Federal Performance Based Planning
The FTA’s Fixing America’s Surface Transportation Act or FAST Act, calls for statewide transportation performance-based planning that integrates standards and targets encompassing every level, national, statewide, regional and local. The FAST Act requires states to consider performance measures and targets when developing policies, programs and investment priorities in the statewide transportation plan. In addition, public transit providers are instructed to set targets, report on progress, develop transit asset management plans and report on the measures. The full description of the federal measures is included in the plan’s Appendix.
MnDOT Performance Measures
MnDOT relies on state based regulations to guide its plans, projects and investments in support of public transit. The GMTIP has four performance measures covering (1) ridership, (2) fleet condition, (3) span of service and (4) on-time performance. The GMTIP identifies targets for each of these performance measures, and MnDOT will annually report progress towards addressing these targets.
MEASURE #1: RIDERSHIP
Increasing public transit ridership is a goal of several MnDOT initiatives and supported plans including the Olmstead Plan’s Measureable Goals, Minnesota Statutes § 174.24, subd 1a, and Heading Home: Minnesota's Plan to Prevent and End Homelessness. The transit ridership performance target is that by 2025 public transit will serve 90 percent of transit needs. To meet this goal, Greater Minnesota public transit must add more service. Ridership is currently tracked monthly and reported annually in the Annual Transit Report. For 2015, MnDOT identified there were 12.1 million transit rides provided, which was approximately 87 percent of the transit need. Progress towards the goal will be reported each year in the Annual Transportation Performance Report and the Annual Transit Report.
MEASURE #2: FLEET CONDITION
This measure is a general indicator of the overall age and accumulated miles of the transit system’s vehicle fleet. It is defined as the percentage of Greater Minnesota transit vehicles that are within their “useful life.” MnDOT’s Asset Management Plan determines the useful life of vehicles.
For transit vehicles, the useful life is based on the combination of miles and years the vehicle has been in service. Each transit system semi-annually reports the age and mileage of transit vehicles. The target is for 90 percent of vehicles to be within their useful life; the minimum threshold is 80 percent. In 2016, 22 percent of vehicles were past their useful life. Future improvements to the public transit systems fleet are needed to meet the minimum threshold for this target. The progress towards the target will be reported each year in the Annual Transit Report.
MEASURE #3: SPAN OF SERVICE
Minnesota Olmstead Plan
In response to a court mandate, the State of Minnesota developed the Minnesota Olmstead Plan, which outlines how state agencies will support individuals with disabilities so they may live, learn, work and enjoy life in the most integrated setting of their choice. The transportation-related goals found in the Minnesota Olmstead Plan were developed by the state and approved by the federal court in June 2015. The goals are designed to remove barriers and improve transportation access to help individuals with disabilities become more independent and integrated into their communities. The span of service plan specifically addresses the core components of increasing access to transit service and connecting employment, housing, services and recreation in Greater Minnesota. The increased service levels will not only increase ridership but play an important role in fulfilling the goals identified in Minnesota’s Olmstead Plan.
Span of Service
This performance measure identifies the percentage of the state’s communities whose public transit span of service meets guidelines established (Figure 7-1). The information is collected using published transit system service schedules. The target is that by 2025, 90 percent of municipalities in Greater Minnesota will have transit service according to their municipal service population. Transit systems shall provide the baseline span of service as local financial resources are available and demand warrants. Currently, only 46 percent of rural and small urban communities meet the weekday span of service guidelines, 4 percent for Saturday service and only one community for Sunday service. The progress towards the span of service will be collected and reported annually in the Annual Transportation Performance Report and the Annual Transit Report.
Figure 7-1: Span of Service
| Municipality Population | Peer Group | Weekday hours | Saturday hours | Sunday hours* |
| 50,000 and over | Urbanized | 20 | 12 | 9 |
| 49,999 - 7,000 | Small Urban | 12 | 9 | 9 |
| 6,999 - 2,500 | Small Urban | 9 | 9 | N/A |
| County Seat Towns* | Rural | 8 (3 days a week) | N/A | N/A |
*As demand warrants based on individual system performance policies
MEASURE #4: TRANSIT ON-TIME PERFORMANCE
Improved reliability is a core component of the GMTIP. Reliability will be measured by on-time performance as the percentage of transit vehicles arriving at their pick-up site within the appropriate window of time. The pick-up window is established in the provider performance standards.
The on-time performance target is 90 percent of trips will be picked-up within the appropriate time window by 2025 and beyond. MnDOT will begin to analyze the data in 2017 and start reporting annually in the 2018 Annual Transportation Performance Report and the Annual Transit Report.
MNDOT EVALUATION CRITERIA
The MnDOT Office of Transit annually evaluates transit system performance to prioritize operating and capital projects. MnDOT ranks each system based on a series of specific criteria and assign each transit system a score. Based on the evaluation criteria, the transit systems are nominally ranked and scores within the bottom 10 percent are targeted for additional technical assistance from MnDOT. Funding allocations are not made based on this information, but does help inform MnDOT about system strengths and weaknesses. The criteria are reviewed and refined annually by the Office of Transit (Figure 7-2).
Figure 7-2: Evaluation Tool Criteria
| Evaluation Criteria | Access | Performance | Financial Metrics |
| Factors | - Percentage of Countywide Needs (hours per capita) - Minimum Level of Access - Rural Service Volume - Span of Service | - Passengers per Hour - Percentage of Service in underperforming routes - On-time performance | - System Revenue to Expense - Cost Per Trip - Cost Per Service Hour |
Transit System Guidelines and Standards
- No related sections.
OVERVIEW
MnDOT has a strong commitment to support efficient and effective public transportation services in Greater Minnesota. The guidelines and standards presented in this section reflect the six types of service that are operated by Greater Minnesota public transportation systems (Figure 7-3). Each service type has guidelines and standards to help the system track their own performance (Figure 7-8 and 7-9).
Figure 7-3: Six Service Types
| Service Type | System Type | Application |
| Fixed-Route | Urbanized (5307) | Urbanized Communities |
| ADA Complementary Paratransit | Urbanized (5307) | Urbanized Communities |
| Dial A Ride Demand Response | County-wide and Regional Multi-county systems | Non-Urbanized Communities |
| Deviated Route Demand Response (Municipal) | ||
| Deviated Route Demand Response (Rural/Regional | ||
| Intercity Bus Feeder | County-Wide and Regional Multi-county systems (5311 F) | Urbanized and Non-Urbanized Communities |
Fixed route service - a vehicle is operated along a prescribed route according to a fixed schedule.
ADA complementary paratransit - Transportation service required by the Americans with Disabilities Act (ADA) for individuals with disabilities who are unable to use fixed route service. This service must be comparable to the level of service provided to individuals without disabilities who use the fixed route. The complementary services must be origin-to-destination service (demand response). Service must be provided in a corridor three-fourths of a mile on either side of the fixed route.
Dial-a-ride service - operates in a defined area such as a city, county or transit agency jurisdiction during advertised days and hours. Pickups and drop-offs typically take place anywhere within the service area and sometimes at important out-of-area locations. Riders call to request a pickup time and service providers develop schedules and routes according to these requests.
Route deviation service - operates along established routes that typically have designated stops. Between these stops, vehicles deviate from an established route to pick up or drop off riders within a defined off-route service area.
SYSTEM GUIDELINES AND STANDARDS
Throughout the GMTIP planning process, MnDOT identified 24 metrics in collaboration with Greater Minnesota transit providers. MnDOT highly recommends each system choose, adapt and refine some of the proposed guidelines to reflect the operational characteristics of each system. In addition to tracking performance, should a system ever reduce service, the reductions should be based on performance standards to comply with Environmental Justice requirements.
MnDOT highly recommends, but does not require the transit systems to adopt these measures. Performance measurement is a good business practice. Using performance measurement transit systems can; identify problem areas for further analysis, generate information for policy formulation, measure goal attainment for priority areas, and determine resource allocation.
The benefit of implementing the metrics is that each system will develop a stronger profile of their system’s strengths and weaknesses. The metrics are grouped into the following two categories:
1. Service Design and Reliability Guidelines: (not associated with cost or productivity)
- Service Design Guidelines (1) facilitate access to high-quality public transportation (service frequency, and service hours per capita) and (2) provide multimodal amenities and safe waiting areas (bicycle parking at transit stops, continuous walking routes, and crossings to stops)
- Reliability Guidelines (1) provide convenient and reliable service (on-time performance and advanced reservation time) and (2) maintain fleet to ensure passenger safety and state of good repair (road calls, accidents, and spare ratio)
2. System Performance Standards (related to cost or productivity):
- Ridership: Measure the change in network usage (passengers per hour) and ensure services operate responsibly (cost per ride).
SERVICE DESIGN AND RELIABILITY GUIDELINES
The Service Design Guidelines are intended to guide allocation of transit resources and to work towards regional coordination and consistency. These guidelines represent the general types of transit service, enhancements and amenities that are appropriate to implement; however, exceptions exist based on local circumstances and funding. Use of these service design guidelines is optional, but highly recommended. MnDOT will not mandate use of the guidelines or require new reporting. These measures are to benefit the transit systems for their own reporting and use.
Provider Reliability Guidelines
Reliability of transit service has been recognized as a significant determinant of quality of service in the plan. The reliability guidelines are intended to serve as indicators for the transit agency to measure and monitor. These guidelines are representative of the general performance thresholds service that systems may attempt to reach. However, exceptions often exist based on specific local circumstances and available funding.
System Performance Standards
Performance standards evaluate the productivity and efficiency of services provided. To be responsible and dynamic, a transit system must consistently measure and adjust service accordingly. These standards serve as indicators of route performance and call attention to routes that may need adjustment. The use of multiple performance standards provides better insight into the operational and financial performance of services and allows transit providers to balance the cost and ridership of each route in the system’s service network.
The examples below, passengers per hour, passengers per trip, cost per passenger and cost recovery describe the basic concept and why the information is valuable to collect.
Productivity: Passengers per Hour
Productivity is measured as the number of passengers per hour (Figure 7-4). Productivity is calculated by the total number of passengers carried divided by the total service hours. A high number of passengers per hour show a route is serving more people. The passengers per hour metric is calculated at both the route and trip level, but can be also viewed on a per bus basis to establish a minimum standard of route performance. Figure 7-4 shows the minimum passengers per hour. Passenger per hour is applicable for all service types and in all communities.
Figure 7-4: Productivity: Passengers per Service Hour
| Service Type | Route Average* |
| Fixed Route | 15 |
| Commuter Bus | 15 |
| Route Deviation (Urban/Community) | 8 |
| Route Deviation (Rural) | 5 |
| Dial A Ride (Urban/Community) | 3 |
| Dial A Ride (Rural) | 2 |
| * Route average represents the average passengers per service hour over the entire day. Individual hours may fall below the standard. Also, service hours is defined as one bus operating for one service hour. | |
For example, Route 1 operates three buses; each operates eight hour per day. The daily ridership on Route 1 is 96. The route productivity average is four passengers per hour
Productivity: Passengers per Trip
The passengers per trip applies to intercity and regional mobility services only. These services are typically several hours in length. Therefore, the standard for passengers per hour does not apply. This standard describes the minimum acceptable capacity of service operating on a given route. Routes that do not meet these minimum standards should be reviewed for potential changes to increase ridership or reduce service. Very poor performing routes may be considered for elimination.
Figure 7-5: Passengers per Trip
| Service Type | Minimum Per Trip |
| Regional Mobility | 3 |
| Intercity Bus Feeder | 3 |
Cost Effectiveness: Cost per Passenger
A route’s cost effectiveness is measured by the cost required to deliver service on a per passenger basis. This standard identifies the possible cost ranges when comparing overall system averages and focuses on corrective action for those services falling below average. Figure 7-6 shows the cost per passenger thresholds and possible corrective action. Routes should be assessed after being in operation for one year.
Figure 7-6: Cost per Passenger
| Cost Per Passenger | Monitoring Goal | Possible Action |
| 20 to 35 percent over system average | For quick review | Minor modification to route |
| 35 to 60 percent over system average | For intense review | Major changes to route |
| Greater than 60 percent over system average | For significant change | Restructure or eliminate route |
Cost Effectiveness: Cost Recovery
A second measure for determining route cost effectiveness is the percentage cost recovery for a route (revenue/expense). Cost recovery calculates the amount of revenue generated by a service to cover the operating expense. Revenue typically includes fares, contract revenue, local contributions or local tax subsidy.
MnDOT recommends transit systems generate a minimum of five percent excess revenue on their services (20 percent rurals/25 percent urbanized). By increasing the revenue beyond the amount needed to pay the local share for the service (15 percent rurals/20 percent urbanized), the excess revenue is available for capital match or match on service expansions that do not have a revenue source for the local share.
Implementation of a Performance Monitoring Framework
Establishing a set of transit performance guidelines helps evaluate the adequacy of existing transportation services provided by Greater Minnesota public transit systems and guides the development of proposals that improve those services. Initially, these guidelines should be used as a baseline as each system defines its own set of standards associated with its appropriate service type.
Several points should be made with respect to the development and subsequent application of the performance guidelines. First, reasonable judgment must be used in applying the guidelines to assess the current service. While guidelines are quantitative for the most part, unusual situations may arise which warrant special consideration. Issues related to public policy and funding cannot always be addressed fully by numerical guidelines.
Second, the guidelines may conflict since some relate to the benefits to be derived from transit service while others relate to their costs. Nonetheless, the guidelines permit the tradeoffs to be defined and an informed decision made to resolve differences.
Third, the comparison of actual performance with the guidelines should not be made on a “pass fail” basis. Instead, results should be viewed in terms of the proportion of time the guideline is met or the level of attainment. Finally, the guidelines have been set at reasonable values that can be achieved or that can serve as useful “targets.”
MnDOT recommends transit agencies use a five-step process to adopt the guidelines.
- Identify the service types the system operates. (fixed route, deviated route, etc.)
- Initially, providers should select only a few of the 24 potential metrics.
- Determine the performance of the system for the metrics that are selected. For each transit system, both the system-wide and individual performance of each service should be considered.
- Identify the guidelines and standards for the system associated with the metrics being reviewed.
- Review system performance for the applicable service type and metrics listed in the guideline. Based on the review, set the standards to reflect the system’s particular situation.
The periodic application the guidelines and standards can become a powerful tool in guiding the restructuring of services to productivivity and better serve residents. Transit systems should review service standards every few years to determine whether the standards should be revised to reflect changes that have been made. For example, a recent trend of mergers among Greater Minnesota transit systems may result in differences as to the manner in which an organization might interpret the guidelines or standards differently prior to or after the merger’s effective date.
Service Design Guidelines
Service Design Guidelines are intended to guide the appropriate allocation of transit resources and ensure regional coordination and consistency. These guidelines are representative of the general types of transit service and service enhancements and amenities that are appropriate to implement. However, exceptions often exist based on specific circumstances and available funding. Figure 8-8 shows the recommended guidelines for service design.
Reliability Guidelines
Reliability of transit service has been recognized as a significant determinant of quality of service in the plan. The reliability guidelines are intended to serve as indicators for the transit agency to measure and monitor. These guidelines are representative of the general performance thresholds service that systems may attempt to reach. However, exceptions often exist based on specific local circumstances and available funding. Figure 7-7 shows the recommended guidelines for reliability.
Figure 7-7 Provider Service Design Guidelines
| Metric | Fixed route | Route Deviation | Dial a Ride | Regional Mobility | Commuter Bus | Intercity Bus Feeder | Vanpool |
| Service Hours: Span of Service | These three types of service are to be provided according to the baseline span of service. Type provided as demand warrants. | ||||||
| Service Frequency | - 60 minutes or better - 30 minutes or better peak hours | - 30 minutes or better w/o DAR, 60 min or better with DAR | 2 round trip per week | - Minimum 2 round trips in morning, 2 round trips in afternoon - Peak: 30 to 60 minutes - Midday: At least one round trip if market supports | -3 round trips per week | ||
| Service availability: % of population who have local transit service available | 75% of the service area population within 1/4 mile of a transit route | 75% of service area population within 3/4 mile of a transit route | 75% of population covered by service area | - 80% of communities in service area have regional mobility service - 80% of the service area population have regional mobility service | 80% of population within 25 miles of intercity bus stop | 4 - Maximum number of pick-up locations 2-4 - maximum number of drop-off locations | |
| Service hours per capita | 2.0 | 0.45 | 0.45 | ||||
| Information availability (print, online, translated) | Standard requirements: - Title VI, Riders Guide, Service Schedules (Locations/time), trip reservation process | -Publicly advertise the availability of route deviation service. -Publish deviation policy/procedure. All other standard requirements | Standard Requirements: - Title VI, Riders Guide, Service Schedules (Locations/time), trip reservation process | ||||
| Planning Requirements | Urban areas over 50,000 - Identified and analyzed as part of Transit Development Plan | Meets public participation requirements. Service expansions must be determined through an alternatives analysis. | - Meets public participation requirements - Service expansions must be determined through an alternatives analysis. | ||||
| Number of shelters installed | Shelters at stops with at least 20 boardings per day or major transfer points | Shelters at stops with at least 15 boardings per day or major transfer points | |||||
| Bicycle paring present at transit stops | Bike parking at stops with at least 20 boardings per day | Bicycle Access on Buses | Bike Parking at stops with at least 20 boardings per day | Bicycle Access on Buses | |||
| Continuous walking routes and crossings to stops | Pedestrian Facilities within 1/4 mile of stops with at least 20 boardings per day | Pedestrian facilities within 1/4 mile of stops with at least 20 boardings per day | |||||
| On-time performance | 90% of schedule stops on-time, within 5 minutes after a scheduled stop | No bus shall depart a formal time point before the time published in the schedule. 90%- on time performance | 90% on time within published pickup window. Urban Window – 20/20 minutes Rural Window – 45/45 minutes. | Should always depart on-time, notice should be provided to riders in unusual weather circumstances | Should always depart on-time, notice should be provided to riders in unusual weather circumstances | Should always depart on-time, notice should be provided to riders in unusual weather circumstances | 90% of schedule stops on-time, within 5 minutes after a scheduled stop |
| Advance Reservation Time | For deviation requests: - Urban - Minimum 2 hrs. in advance - Rural - Minimum 24 hours in advance Next day service | -Urban - Minimum 2 hrs. in advance - Rural - Minimum 24 hours in advance Next day service | |||||
| Reservation Negotiation Window | Maximum: Up to an hour before or after requested time | ||||||
| Trip Denials | Transit systems must follow the ADA trip denial definitions and process | ||||||
| Trip Cancellations | Bus or vanpool trips should only be canceled from lack of riders or weather cancelations | ||||||
| Passenger complaints | The benchmark is 6 complaints/100,000 boardings. | A formal process should be established for resolving problems/complaints | |||||
| Road calls | The benchmark is 1 road call/14,000 revenue miles. | Should be serviced (oil change and other preventative) maintenance every 7,500 miles. | |||||
| Accidents | Fewer than 1 recordable accident per 100,000 revenue miles | ||||||
| Fleet Maintenance | At least 75% of all regular fleet vehicles should be available for operations at all times. | ||||||
| Spare ratio | The ratio of spare vehicles to regular fleet vehicles should be less than 20% | Vanpool providers should be able to secure a spare vehicle within one business day | |||||
| Cost per ride | $5.00 | $6.00 | $15.00 | ||||
| Cost recovery (% of operating cost) | 15% (includes local subsidy) | 15% (includes contract revenue and local subsidy) | 15% (includes contract revenue and local subsidy | 15% (includes local subsidy) | 25% | 15% | 50-100% (< $ 120 monthly cost to vanpool user) |
| Passengers per service hour | 15 | 8 | 3 | 3 boardings per trip | 15 | 3 boardings per trip | |
Sections in Chapter