A SAMPLE REPORT. Benchmarking to Improve An Examination of Districts Processes in: INFORMATION TECHNOLOGY (IT)

PIIE

Process Improvement and Implementation in Education

A SAMPLE REPORT

Benchmarking to Improve

An Examination of Districts’ Processes in:

INFORMATION TECHNOLOGY (IT)

A national study based on sur veys from the following Districts:

Aldine, TX Anne Arundel, MD Atlanta, GA Boston, MA Brazosport, TX Broward County, FL

Carrollton Farmers Branch, TX Cobb County, GA Montgomery County, MD Pajaro Valley, CA Philadelphia, PA Pinellas County, FL Pittsburgh, PA Tulsa, OK Wake County, NC Elk Grove, CA Fairfax County, VA Galena Park, TX Galveston, TX Gwinnett County, GA Houston, TX Los Angeles, CA Miami Dade County, FL

P I I E

A preliminary performance comparison of how their processes in

MANAGING INFORMATION TECHNOLOGY

compared to processes in other participating districts

T

C

About this Report . . . 2

What’s in this Report . . . 3

Section 1. Quantitative Data . . . 3

Section 2. Qualitative Data . . . 3

Section 3. Appendices . . . 3

Important Caveats About the Data . . . 4

Success Factors and Challenges . . . 6

Success Factors . . . 6

Challenges . . . 6

Section 1. Quantitative Data . . . 7

Your Performance Scorecard Explained . . . 7

Your Performance Scorecard . . . 9

Cost Effectiveness . . . 12

Staff Productivity . . . 15

Process Effi ciency . . . 18

Cycle Time . . . 23

Section 2. Qualitative Results . . . 31

Section 3. Appendices . . . 37

Appendix A: Characteristics of Participants . . . 37

Appendix B: Identifying Top Performing Academic Districts . . . 39

Appendix C: APQC Process Classifi cation Framework . . . 41

Appendix D: About APQC, OSBC Research, and PIIE . . . 45

Appendix E: Glossary . . . 47

A

R

This is a report comparing your district’s performance with other participating districts in a process called Managing Information Technology (IT).

This report is part of a national pilot project called PIIE—Process Improvement and

Implementation in Education. Data in this report were collected through the fi rst version of a survey that was blinded and analyzed. Finally, a customized report was prepared for your district comparing your performance in this particular process to that of other participating districts. Since this is a pilot project, the fi ndings are preliminary and may fl uctuate in future versions based on larger samples sizes and modifi cations to defi nitional elements to terminology. Special attention will be made to work with the pilot districts to refi ne the survey instrument to ensure that it will allow districts to more accurately answer questions. Although a signifi cant amount of time was spent validating survey responses, many districts were unable to obtain data that could encompass the entire survey scope, and thus it will be an ongoing process to help refi ne responses.

This survey was part of a year-long pilot conducted by APQC, a 30-year-old nonprofi t

organization located in Houston, Texas. APQC’s purpose is to help any organization to improve whether it’s in education, business, health care, or government. Valid comparisons can be made because every participant uses a common taxonomy of processes and sub-processes called the APQC Process Classifi cation FrameworkSM (PCF), explained in later pages.

How can you use this report? The analysis of your data focuses on four types of

measurements or key performance indicators (KPIs). • Cost Effectiveness

• Staff Productivity • Process Effi ciency • Cycle Time

In addition, this report contains “Key Findings” reported by the districts. The analysis that is captured in this report will allow you to see where your district compares with others. Combined with your knowledge about your district’s strategic objectives and challenges, the data from the report can help you to focus your improvement activities to align to your district’s goals.

We strongly believe that this report indicates signifi cant opportunities for process improvement and innovation in every district. Please let us know if we can help your district make these improvements.

C. Jackson Grayson, Jr. Diane Kline Travis Colton

W

’

R

Section 1. Quantitative Data

This section is divided into four categories of performance metrics that are listed on the following page. Each metric within the category has a graph that shows your district compared against all other participants, districts of similar size (over/under 100K students), and a select group of districts that have been identifi ed as high academic achieving, shown as High Achievement on the charts (see Appendix B for details). All charts show the median values for each group.

Section 2. Qualitative Data

Features questions, responses, and patterns related to non-metric variables from the survey. Comparisons are made between your district and all participants, high-achieving districts, and districts of similar size. Not all questions from the survey are shown: only those that showed interesting patterns that might lend themselves to further investigation.

Section 3. Appendices

• Background information about the demographic composition of the survey participants

• Methodology for identifying top performing academic districts

• Description of the Process Classifi cation Framework (PCF), which the metrics and processes for this survey are based on

• An overview of APQC and the PIIE Project

• Condensed glossary with IT and benchmarking terms

I

C

A

D

As you read this report, keep the following things in mind as you evaluate your performance compared to the current PIIE database.

1. Pilot data. These are pilot data from 22 districts of varying sizes. Since this is a

pilot project with a small sample size, the data depends highly upon the accuracy of each district’s data supplied, in terms of their ability to supply accurate data and a common understanding of the scope of the survey questions. While this data has been statistically verifi ed by APQC, in some cases the metrics will shift after further evaluation with individual districts. It is critical to note that outlying data points have been removed where appropriate to improve overall accuracy. It is also critical to note that this type of metric fl uctuation is normal and will be eliminated as new districts fi ll out the surveys and are added to the PIIE database.

PIIE will continue to review and revise the data collection surveys, process defi nitions, and comparisons. As time goes by, we will be adding hundreds of additional districts and the data quality and comparisons will become more stable and reliable.

2. Low and high deciles. Be careful how you interpret and use the comparative

data, regardless of whether it is in the low or high deciles range. You may have valid reasons for your applicable metric to be where it is – your history, your goals, and your strategy. Every district needs to examine the data in light of their own situation and evaluate their own opportunities for improvement. PIIE can’t do that. The deciles comparison depends on what is being measured and how the measure relates to outcomes. Let’s say your district spends more on assessment per

student. Your district may be spending more than other districts but you may also be getting better results, have different strategies, or have just had a huge capital investment.

3. Correlation and Causation. Simple correlations do not prove causation. There

may be many factors causing differences in costs, quality, cycle time, etc. As more data is collected, we plan to look for causal linkages through multi-factorial analysis. That said, correlations can be useful directional signals now about where to look deeper for improvement. Do these caveats mean that the data the pilot has collected, analyzed, and compared are useless or can be ignored? Not at all.

First, participating districts now know more about their own process data than they did before, especially cross-functional process data.

Two, they also now know more about how other districts are spending time or money on selected processes, enabling them to examine their own processes more deeply for improvement opportunities, and what and who they need to benchmark with.

Three, PIIE has also learned much about district data systems and ways in which surveys can be improved.

Success Factors

The following are specifi c processes and activities that districts cited as key enablers to their success. The fi rst column comes from districts identifi ed as high achieving, and the second column lists common factors/challenges across all participants.

S

F

C

HIGH ACHIEVING DISTRICTS ALL PARTICIPANTS

Rapid response to customer change requests as well as the drastic shifts in technology.

Strong leadership, specifi cally, leadership that has “strategic vision for the entire organization and how IT fi ts into the organizational goals.” Clearly defi ned goals and objectives that

promote effective communication with IT stakeholders.

Effective communication is reported by 32% of the school districts as a measure for successful IT initiatives.

Centralized management and resources. Adequate funding, especially for the migration from legacy systems.

Stable technology framework, lending itself to zero unplanned down-time.

Consistent staff IT training.

Challenges

These are commonly listed obstacles that hinder effective management of the various IT processes throughout the participating districts.

CHALLENGES

Seventy-seven percent listed lack of funding as the biggest challenge encountered in

implementing and maintaining information technology. Limited budgets were shown to impact many areas including maintaining and replacing hardware, updating software, acquiring skilled staff members, and training existing staff.

Fifty-eight percent claimed having diffi culty keeping up with the rapid changes in technology and the subsequent increase in demand for technology services “[It is a challenge] supporting growing 24 hours a day, 7 days a week, 365 days a year technology services for students, faculty, parents, and the community with an 8-5 staff.”

Thirty-seven percent reported that technology is still not viewed as a high priority by their management personnel.

A reported 32 percent of participating school districts are having diffi culty keeping their staff fully trained in respect to technology.

“Silos, which are [the main] organizational structures of public school systems [are a stumbling block for effective communication].” Twenty-six percent listed the challenge of getting people and departments to communicate.

Your Summary of Key Indicators Explained

On the following pages (8–9), you will see a spreadsheet called Summary of Key Indicators. It lists both key performance indicators (KPIs) and supporting indicators with columns that show your district’s responses alongside those of all participants.

The indicators are divided into the following four categories:

• cost effectiveness

• staff productivity

• process effi ciency and

• cycle time.

The performance measure for the indicators is compared against all participants. Your Summary of Key Indicators uses the following labels:

N

A count of the number of districts who provided data to calculate the metric.

Low 20th

This term refers to the 20th percentile of the distribution of the metric on the low side of the median. It does not necessarily infer directional goodness of the data.

Median

This refers to districts performing at the median.

High 20th

This term refers to the 20th percentile of the distribution for the metric on the high side of the median. It does not necessarily infer directional goodness of the data.

Lower Bound and Upper Bound

The lower and upper bounds represent the range of values that fall within the 95% confi dence interval. Meaning that we can be 95% confi dent that all districts will fall within this range. If your district’s measure is outside of this range, it may be an indicator of two main issues. First, it may mean that your response(s) did not meet the scope of the survey questions and need to be reviewed. Secondly, if your response(s) are correct it may show areas where your district should validate

your current processes to understand why you are performing outside the range. It may be associated with higher performance or areas of needed improvement. Further PIIE research will begin to identify which direction higher performance is associated with.

The comparison of your district’s metrics to the overall metric distribution can be used to detect patterns of how top achieving districts are doing in specifi c areas. Your comparisons can be used in context with your district’s goals to improve where you see necessary.

Cost Effectiveness Your District N 20th Percentile Median 80th Percentile Lower Bound Upper Bound

Total IT Budget per

Student $305 18 $160 $266 $491 $143.15 $389 Total IT Budget as a Percentage of Total District Revenue 2.5% 18 2.3% 2.7% 5.0% 1.5% 3.9% IT Budget per Employee $1,925 17 $1,232 $2,116 $3,423 $1,110 $3,122

All Participants 95% Confi dence Range

Your Performance Scorecard

Staff Productivity Your District N 20th Percentile Median 80th Percentile Lower Bound Upper Bound

Customers per IT FTE 479 18 475 756 1,225 407 1,106

Computer Units per

IT FTE 536 18 233 540 868 291 790 Process Effi ciency Your District N 20th Percentile Median 80th Percentile Lower Bound Upper Bound IT FTEs per $100M District Revenue 25.6 22 12.3 18.3 28.2 10.6 25.9 Percentage of Unscheduled Outages from Change Requests

4.3% 19 0.1% 1.0% 0.0% 0.6% 1.4%

Percentage of Unscheduled Outages from Release

Instructions

1.6% 20 0.9% 1.5% 9.0% 0.8% 2.2%

Percentage of Data and Content Managed

as Enterprise Information Assets (Centrally Managed)

Your Performance Scorecard

That Are Common Across All Business

Units

61.0% 20 10.0% 77.5% 86.0% 43.5% 100.0%

Total Annual Number of Revisions to the Published Enterprise

Information Architecture

2 15 1.0 1.0 3.2 0.5 1.5

All Participants 95% Confi dence Range

Cycle Time _DistrictYour N 20th

Percentile Median 80th Percentile Lower Bound Upper Bound

Average Project Life Cycle in Months for Major IT Projects or

Initiatives

14.0 21 6.0 12.0 18.0 6.9 17.1

Average RFP Life Cycle in Months for Major IT

Projects or Initiatives

5.9 21 3.0 4.0 6.0 2.3 5.7

Average Time in Months to Fulfi ll the IT Needs of a Major Business Shift

8.4 20 5.8 12.0 16.0 6.7 17.3

Your Performance Scorecard

Average Time in Weeks to Fulfi ll a Business

Information Need

9.5 22 7.0 7.0 14.0 4.1 9.9

Average Time in Weeks to Fulfi ll a Student Information Need

8.5 22 0.0 7.0 7.0 4.1 9.9

Average Time in Weeks to Create the Enterprise Information Management Strategic Plan 7.0 16 6.0 12.0 13.0 6.1 17.9 Planning Horizon for Information Architecture in Years 5.0 19 3.0 4.0 5.0 2.2 5.8

Time in Weeks to Report on Compliance Status

of the Information Architecture

3.4 9 1.0 2.0 2.8 0.7 3.3

Time in Weeks to Make a Change into the Production Enviornment

2.0 20 1.0 2.0 4.0 1.1 2.9

Average Time in Weeks to Introduce a New Release into the Production Environment

8.0 20 2.0 4.0 12.0 2.2 5.8

Average Time in Hours to Resolve a Service Commitment Disruption

1.7 21 2.0 4.0 6.0 2.3 5.7

Cost Effectiveness

Cost effectiveness is a ratio between how much your district spends per unit of measure, such as the information technology (IT) budget per student. Essentially, these measures allow you to compare your expenses against those of other

districts in a way that only controls for factors such the number of students and the district’s overall revenue.

Total Information Technology (IT) Budget per Student

Total IT budget per student includes both operational and capital budgets specifi cally associated with IT processes. Typically, cost measures use cost/ expenditures, but most districts do not track their budgets on an activity-based level, making it extremely diffi cult to gather accurate budget fi gures.

Regarding district IT budget calculations, the scope of this study is to include all IT-related budgeted costs. In some cases, districts may not know or not include all components of their budget value. In other cases, districts may provide IT services other than to students. For example, a district may conduct IT development for products licensed to other districts. In this case, their IT budget would refl ect non-student services, thereby infl ating this value.

$210 $201 $0 $100 $200 $300 $400 $500 $266 $305 High Achievement Your District Size < 100K All N = 18 All Size < 100K High Achievement Your District

z

zz

KEY FINDING FOR COST EFFECTIVENESS

IT budgets vary widely among districts. Of the 22 responses given to the IT survey,

the IT budgets ranged widely when normalized per student. This may be attributed to a district’s capability to measure process-related IT budgets at the activity level.

Student achievement does not appear to be highly correlated with IT spending.

While the data is preliminary, the top performers in terms of student achievement actually spent less per student than the median. This may also mean that the top-performing achievement districts are less likely to measure all the IT processes that make up the entire IT budget.

A relating fi nding applies to district IT budgets as compared to overall budgets. The top achievement districts have virtually the same IT budget percentage compared to the overall district budget.

Regarding the relationships between IT budgets per student, district size, and free/reduced lunches, almost no difference exists between the size of the district, their levels of qualifi ed free/reduced lunch participation, and the amount spent per student. As a result, the amount of IT spending does not appear to be highly correlated to the overall student performance within a district. The primary difference seems to lie in how the top performing districts are using their budget.

The top performing district peer group has a slightly lower IT budget per employee.

Of the top performing achievement group, their yearly IT budget per employee was lower ($1,884) compared to the overall group ($2,116).

The top performing district peer group spends a higher percentage of their IT budget on IT growth opportunities. The top performing achievement group spent,

on average, 5 percent more of their total IT budget on projects that they categorized as IT growth in lieu of IT operational needs. This refl ects in general the willingness of the districts that spend more on growth to pursue technical solutions that provide advantages where necessary.

Median IT Budget per Student (Yearly)

Top Student Achievement Performers $201

All Other Participants $266

IT Budget/Overall Budget (Yearly)

Top Student Achievement Performers 2.8%

IT Budget as a Percentage of Total District Revenue

This cost measure compares the planned overall IT budget for the most recent year divided by the overall district budget for the same time period.

Total IT Budget per District Employee

This cost measure gauges the blended average of IT budget across all district employees (both instructional and non-instructional) serviced by the IT function.

2.4% 0.5% 1.0% 1.5% 2.0% 2.5% 2.7% 2.5% High Achievement Your District Size < 100K All All Size < 100K High Achievement Your District N = 18 0.0% 3.0% 2.8% $1,884 $500 $1,000 $1,500 $2,000 $2,500 $1,925 $2,116 High Achievement Your District Size < 100K All All Size < 100K High Achievement Your District N = 17 $0 $3,000 $1,963

Staff Productivity

Staff productivity indicators measure the ratio of input to output, both of staff and the IT management processes as a complete system. The two metrics within this category attempt, at a macro level, to show the workload of the IT staff. At this level of measurement, they include all aspects of managing IT activities, extending from strategic planning at the executive level to customer support at schools.

Computer Units per IT Full-Time Equivalent (FTE)

This productivity measure gauges computer resource support for each IT FTE on a general basis. It includes both dedicated as well as shared PCs and laptops for students, teachers, and non-instructional staff.

501 658 0 100 200 300 400 500 540 536 High Achievement Your District Size < 100K All N = 18 All Size < 100K High Achievement Your District 600 700

z

zz

KEY FINDINGS FOR STAFF PRODUCTIVITY

More customers are supported per FTE in the top performing achievement group. The ratio of customers (including students, teachers, non-instructional staff, parents, and others) to IT full time equivalents is higher for the top performing achievement group, indicating that fewer IT resources are needed per customer to support their customer base. This can be attributed to a possible combination of more effi cient support processes and alternative district staffi ng priorities. It can also be attributed to a district’s specifi c activity-based measurement capabilities (or lack thereof).

On a related note, more computers are supported per IT support FTE in the top performing achievement group. The ratio of computers (including both desktops and laptops for all dedicated student computers, dedicated teacher computers, shared computers, and non-instructional computers) to FTEs was 658 for the top-performing achievement districts and 540 for all participants.

The top performing achievement group spends nearly twice as much on IT professional development. Note that this fi nding carries as much weight regarding cost effectiveness as it does for staff productivity. Top performers spend nearly $1,300 annually per IT FTE on IT professional development compared to $650 for all districts. IT resources, no matter what district they work for, require ongoing pertinent technical training. The top-performing districts fi nd ways to allocate additional funds for pertinent IT professional development.

Customer: IT FTE Ratio

Top Student Achievement Performers 1,022 customers per FTE All Other Participants 756 customers per FTE

Customers per IT FTE

Similar to units per IT FTE, this productivity measure is a general gauge of the number of IT customers (students, teachers, and non-instructional staff) serviced by the IT function divided by the total number of IT FTEs. All other variables being held constant, a more productive system (both staff and processes) per IT employee will allow a district to provide IT services to a larger number of customers with the same or fewer number of staff resources.

719 1,022 0 200 400 600 800 1,000 756 536 High Achievement Your District Size < 100K All N = 18 All Size < 100K High Achievement Your District 1,200

Process Effi ciency

Process Effi ciency indicators gauge the overall effectiveness of a group of processes in terms of data centralization and standardization, levels of responsiveness, and reduction of time spent performing unplanned activities as well as staffi ng gauges intended to give resource estimates to achieve these indicators.

IT FTEs per $100 Million of Total District Revenue

This effi ciency measure compares the overall total number of IT employees to the overall district budget for the most recent year.

20.2 17.5 0 5.0 10.0 15.0 20.0 25.0 18.3 25.6 High Achievement Your District Size < 100K All N = 22 All Size < 100K High Achievement Your District 30.0

zzz KEY FINDINGS FOR PROCESS EFFICIENCY

The top achievement peer group has a slightly lower ratio of IT FTEs per $100 million in district funding. When normalized by total budget, the top-performing

achievement districts employ slightly fewer IT FTEs per total budget dollar than the general district population. The ratio of IT FTEs to total district budget is 17.5 for top-performing achievement districts, and 18.3 for the general district population. When combined with the fi ndings regarding staff productivity, the top achievement districts are able to support more customers per IT FTE and have a lower IT

resource requirement per district funding dollar.

Centralization and standardization of data affects process effi ciency. When

searching for reasons for improved effi ciencies by top performing districts, two likely areas to explore are the levels of data standardization within districts and the level of centralization of the data. Slight differences in these metrics appear in how districts consider enterprise data assets and the commonality of these data assets across all sites.

Two reasons for the difference in the customer service ratio per IT FTE may be the differences in how districts consider enterprise data assets and the commonality of these data assets across all sites. The differences outlined below can be attributed to improved IT effi ciency in terms of acquisition, maintenance, and accuracy of enterprise data assets due to efforts to standardize and centrally store student data for analysis purposes.

Improving the level of standardization of data across a district as well as improving the level of accessibility of the data for analysis purposes can improve the effi ciency of the IT organization by reducing turnaround time for data requests and will also enrich the quality of data for better analysis for such efforts as disaggregating student assessment data.

Percentage of Information Assets that are Common Across District Sites

Top Student Achievement Performers 85.0% All Other Participants 77.5%

Percentage of Centrally Managed Enterprise Information Assets

Top Student Achievement Performers 90.0% All Other Participants 88.0%

Percentage of Unscheduled Outages from Change Requests

This effi ciency measure looks at the percentage of unplanned or unscheduled interruption in service availability due to implementation of a change request. For example, the transportation offi ce may ask that the bus routing system be modifi ed to allow substitute driver names to be added to a routing screen.

Percentage of Unscheduled Outages from Release Introductions

This effi ciency measure looks at the percentage of unplanned or unscheduled interruption in service availability due to implementation of a release introduction. An example of a release introduction is an updated version of a student

information system that signifi cantly modifi es the functionality of the application.

1.0% 1.5% 1.0% 4.3% High Achievement Your District Size < 100K All N = 19 All Size < 100K High Achievement Your District 0.0% 1.0% 2.0% 3.0% 4.0% 5.0% 2.0% 6.0% 1.5% 1.6% High Achievement Your District Size < 100K All N = 20 All Size < 100K High Achievement Your District 0.0% 1.0% 2.0% 3.0% 4.0% 5.0% 6.0% 7.0%

Percentage of Data Managed as Enterprise Information Assets

This effi ciency measure shows the percentage of data and content that is managed by centrally controlled enterprise assets or systems such as a student information system. 90.0% 90.0% 88.0% 74.0% High Achievement Your District Size < 100K All N = 21 All Size < 100K High Achievement Your District 0.0% 10.0% 20.0% 30.0% 40.0% 50.0% 60.0% 70.0% 80.0% 90.0% 100.0%

Percentage of Information Assets That Are Common Across All Units

This is an effi ciency measure of the percentage of information assets that are available and/or utilized by all functional areas of the district. An example is the commonality of student data elements across schools.

85.0% 85.0% 77.5% 61.0% High Achievement Your District Size < 100K All N = 20 All Size < 100K High Achievement Your District 0.0% 10.0% 20.0% 30.0% 40.0% 50.0% 60.0% 70.0% 80.0% 90.0% 100.0%

Annual Revisions to the Published Enterprise Information Architecture

This process effi ciency measure looks at the total number of revisions to the district’s offi cial enterprise information architecture plan. This plan is like a blueprint of a building that takes into consideration data structure, hardware, networks, security, etc. and matches it with the overall needs of the district.

Percentage of Information Elements with Data Custodians

A data custodian is typically a middle-level administrator responsible for the collecting and maintaining specifi c data in their functional area, enforcing corresponding policy and procedures, and providing accurate analysis and presentation of data for reporting.

1.0% 1.0% 1.0% 2.0% High Achievement Your District Size < 100K All N = 15 All Size < 100K High Achievement Your District 0.0% 1.0% 2.0% 3.0% 4.0% 5.0% High Achievement Your District Size < 100K All N = 17 All Size < 100K High Achievement Your District 0.0% 10.0% 20.0% 30.0% 40.0% 50.0% 60.0% 70.0% 80.0% 90.0% 100.0% 95.0% 100.0% 80.0% 93.0%

Cycle Time

Cycle time indicators measure how long it takes to perform specifi c IT activities or processes. For most IT organizations, regardless of the industry, these

indicators are typically the most widely measured and utilized to help identify overall performance of the system. It may be acceptable to incur increased costs if it means a signifi cant decrease in the time it takes to resolve customer issues, complete/plan for projects, and implement system enhancements. Holding elements like quality and cost the same, high performance for these measures equates to shorter cycle times.

Average Project Life Cycle in Months for Major IT Projects or Initiatives

This measure indicates the average total elapsed time (including design, develop, implement, test, and deploy) for a district to complete a major IT project. A major IT project is defi ned as requiring more than 320 person hours to complete.

High Achievement Your District Size < 100K All N = 17 All Size < 100K High Achievement Your District 0.0 2.0 4.0 6.0 8.0 10.0 12.0 14.0 12.0 12.0 12.0 14.0

zzz KEY FINDINGS FOR CYCLE TIME

Top performing districts report shorter Request For Proposal cycles on average.

Holding the quality and resource variables constant, the average time to process RFPs for top performing districts is just over three months versus nearly 4.5 months for the general district population. Roughly the same relationship is also found when controlling for district student population. Several factors may relate to this fi nding including the complexity of RFPs, the level of standardization incorporated into the RFP process, the department processing the RFPs, the complexity of the current district solution if an understanding is required by the vendor, and the internal approval process.

Only small cycle time variances exist across districts. This is true for categories

such as service commitment disruptions, responsiveness to business and student information needs, and strategic planning horizons. When dealing with cycle times, top performers were generally only marginally better than the general district population. This indicates a set of base level-of-service delivery cycle time expectations that the IT organization within a district is expected to meet regardless of other technological factors.

Hardware is expected to last on average 4.5 years. Based on the survey responses,

the tightly grouped life expectancy is 4.5 years for desktops and 4.23 years for laptops.

Top student achievement performers tend to have newer equipment versus other participants, by almost 10 percent for desktop units and 4.5 percent for laptops. A larger inventory of modern computers may mean better access to newer software for teachers and students, as well as less downtime due to fewer age-related issues.

Percentage of Hardware Beyond Planned Life Cycle

Desktops Laptops

Top Student Achievement Performers 25.8% 13.0% All Other Participants 33.0% 17.5%

Average Time in Months to Fulfi ll the IT Needs of a Major Business Shift

This cycle time measure indicates an IT organization’s ability to adapt to changing demands on its district. An example of a major business shift is the time to adapt to requirements resulting from a governmental mandate (i.e., No Child Left Behind) or the implementation of an automated system to replace a manual process.

Average Time in Weeks to Resolve an Identifi ed IT Exposure or Opportunity

This indicator measures the amount of time it takes to implement a solution for an identifi ed exposure/risk or an opportunity for improvement. An example is the implementation of a security intrusion detection system.

High Achievement Your District Size < 100K All N = 20 All Size < 100K High Achievement Your District 0.0 2.0 4.0 6.0 8.0 10.0 12.0 14.0 12.0 9.0 12.0 8.4 High Achievement Your District Size < 100K All N = 22 All Size < 100K High Achievement Your District 0.0 1.0 2.0 3.0 4.0 5.0 2.0 2.0 2.9 2.0

Average Time in Days to Fulfi ll a Business Information Need

Typically information needs are ad-hoc requests for data that is stored in applicable system(s). This measures the overall architectural structure and availability of data. An example of a business information need is a request by the grants department to create a report that matches grant funding to actual expenditures at the district and school levels.

Average Time in Days to Fulfi ll a Student Information Need

An example of a student information need is a report request for standardized test analysis or a list of students by school that have moved within the district over the last year. High Achievement Your District Size < 100K All N = 22 All Size < 100K High Achievement Your District 0.0 2.0 4.0 6.0 8.0 10.0 12.0 7.0 7.0 7.0 9.5 High Achievement Your District Size < 100K All N = 22 All Size < 100K High Achievement Your District 0.0 2.0 4.0 6.0 8.0 10.0 7.0 7.0 6.7 8.5

Average Time in Weeks to Create the Enterprise Information Management Strategic Plan

Developing information and content management strategies within the IT

organization includes understanding information needs and the role of IT services for executing the business strategy, assessing the information management implications of new technologies, and planning IT actions and priorities.

Planning Horizon for Information Architecture in Years

This metric indicates the amount of planning a district’s IT organization performs when preparing technology strategies. A longer planning horizon means the district is attempting to account for long-term technology shifts.

High Achievement Your District Size < 100K All N = 16 All Size < 100K High Achievement Your District 0.0 2.0 4.0 6.0 8.0 10.0 12.0 14.0 12.0 8.0 12.0 7.0 High Achievement Your District Size < 100K All N = 19 All Size < 100K High Achievement Your District 0.0 1.0 2.0 3.0 4.0 5.0 6.0 4.0 3.0 4.0 5.0

Time in Weeks to Report on Compliance Status of the Information Architecture

This cycle time metric measures the elapsed time it takes for a district to match the scope and requirements of its current projects and systems with the published information architecture plan. Districts with process maps and systems in place to continually monitor compliance of the current architecture tend to have lower cycle times.

Time in Weeks to Make a Change to the Production Environment

This metric measures the ability of the IT organization to rapidly introduce changes into the production environment (the environment where data/systems are live and available for users to access). It is important to match this metric with effi ciency indicators of downtime to gauge the quality of the change and avoidance of rework. High Achievement Your District Size < 100K All N = 9 All Size < 100K High Achievement Your District 0.0 1.0 2.0 3.0 4.0 5.0 2.0 1.5 2.0 3.4 High Achievement Your District Size < 100K All N = 20 All Size < 100K High Achievement Your District 0.0 1.0 2.0 3.0 4.0 5.0 2.0 2.0 2.0 2.0

Average Time in Weeks to Introduce a New Release into the Production Environment

This is similar to the previous metric, except the introduction of a new release is typically associated with a major change, such as the upgrading of an enterprise fi nancial software package.

Average Time in Hours to Resolve a Service Commitment Disruption

This cycle time metric indicates the responsiveness of an IT organization by measuring the time in hours to resolve disruptions to customers.

High Achievement Your District Size < 100K All N = 20 All Size < 100K High Achievement Your District 0.0 2.0 4.0 6.0 8.0 10.0 4.0 4.0 4.0 8.0 High Achievement Your District Size < 100K All N = 21 All Size < 100K High Achievement Your District 0.0 1.0 2.0 3.0 4.0 5.0 4.0 4.0 4.0 1.7

Average Request for Proposal (RFP) Cycle in Months for Major IT Projects or Initiatives

This measure indicates the average total elapsed time (including design, develop, implement, test, and deploy) for a district to complete a RFP for a major IT project. A major IT project is defi ned as requiring more than 320 person hours to complete.

High Achievement Your District Size < 100K All N = 21 All Size < 100K High Achievement Your District 0.0 1.0 2.0 3.0 4.0 5.0 6.0 4.0 3.3 3.0 5.9

IT Organizational Structure

Districts selected either centralized, decentralized, or hybrid for an organizational structure that best described their IT environment. The chart shows the distribution for all participants, with the vast majority choosing a centralized model.

S

2. B

D

O

D

Percentage of FTEs Performing IT Processes That Are External Service Providers

External service providers (ESPs) offer solutions and services to the internal IT organization but are not part of the organization. Examples include external contractors and consultants.

YOUR DISTRICT: CENTRALIZED

Centralized

71%

A

P

Hybrid

29%

Decentralized

0%

Management of Electronic Information

Districts usually do not struggle with not having enough data to make

data-centered decisions. The challenge is being able to effectively sift through the vast amount of data available and translate it into meaningful information. Linking data sources from disparate data sources into one data warehouse enables knowledge managers and decisions makers to fi nd and report on key information.

Percent of IT Budget Allocated Between Operations and Growth

This measure indicates a district’s expense for operating the existing infrastructure versus the ability to infuse technology innovations into a district. Most of the

budget associated with growth is allocated as capital funds, which is typically used for large projects such as legacy system replacement, network infrastructure, and hardware upgrades. High Academic Achieving Size < 100K All All Linked Most Linked

Quite a Few Linked A Few Linked 0.0% 20.0% 40.0% 60.0% 80.0% 100.0% 36% 32% 27% 5% 60% 20% 20% 50% 25% 17% 8% High Academic Achieving Size < 100K All % Operations % Growth 0.0% 20.0% 40.0% 60.0% 80.0% 100.0% 22% 78% 26% 24% 74% 76%

Distribution of Total Annual IT Operating Budget

The following three charts show the distribution of the IT operating budget for seven major categories.

Software 10%

Hardware 12%

Other 4%

Networks 15%

Facilities 0%

ESPs 7%

Personnel 38%

A

P

>100K S

Y

D

Personnel 38%

ESPs 6%

Facilities 0%

Networks 18%

Other 9%

Hardware 10%

Software 9%

Personnel 50%

ESPs 3%

Networks 10%

Other 10%

Hardware 15%

Software 12%

Currently Utilized IT Methodologies

The following chart demonstrates the prevalence of the following IT methodologies within the responding districts: PMBOK Project Management Body of Knowledge; ITIL IT Infrastructure Library; CobiT Control Objectives for Information and related Technology; CMMI Capability Maturity Model Integration; Other – examples include Software Development Life Cycle (SDLC) and Rapid Application Software Development (RAD). Of the listed methodologies, PMBOK is the most commonly cited.

Currently Utilized IT Methodologies PMNOK ITIL COBIT CMMI OTHER

All Participants 45.5% 40.9% 4.5% 13.6% 9.1% Top Performing—Achievement 30.0% 10.0% 0.0% 0.0% 10.0% > 100K Students 50.0% 40.0% 10.0% 20.0% 10.0%

Your District No Yes No Yes No

A Formal Knowledge Management (KM) Program

The existence of a formal Knowledge Management program for IT suggests the level of capability and standardization of support within an IT organization. The comparison below shows that both achievement and larger districts reported the existence of an IT KM program. Many of the biggest districts have had KM programs for more than 3 years.

50% 50% High Academic Achieving Size > 100K All % Operations % Growth 0.0% 20.0% 40.0% 60.0% 80.0% 100.0% 64% 36% Size < 100K YOUR DISTRICT: YES 50% 50% 75% 25%

Planning for Knowledge Management

Four of the districts that currently have no KM program are in the process of implementing a program for the current or following year. That leaves only 22 percent of the total participants that are not planning on supporting a KM program in the near future. The following chart shows the types of KM programs currently in use.

KM APPROACHES All Participants > 100K Students Your District

Communities of Practice 27.3% 50.0% Yes

After-Action Review 45.5% 50.0% No

Lessons Learned 68.2% 80.0% Yes

Best Practice Transfer 40.9% 60.0% Yes

Expertise Locator Systems 13.6% 20.0% No

PII

PII

Appendix A: Characteristics of Participants

Total Student Enrollment Distribution of Participating Districts

S

3. A

Distribution of Participating Districts by per Student Annual Expenditures

50,000 to 99,999

23%

< 50,000

31%

200,000+

14%

100,000 to 199,999

32%

Annual Expenditures Per Student

10

Distribution of Participating Districts by Locale Type

Distribution of Participating Districts by Students Receiving Free or Reduced Lunch

Mixed Urban

40%

Mid-Size City

5%

Large City

23%

Urban Fringe

32%

Percentage of Total Respondents (n = 22) 10 20 30 40 50 60 70 80 90 2 0 4 6 8 10

Number of Districts

Appendix B: Identifying Top-Performing Academic Districts

While there were several measures that could have been used to determine which districts were most successful in terms of students’ academic achievement, this report focused on a composite of four measures for which most districts reported data, which seemed to provide appropriate comparability across states: average

advanced placement (A.P.) test scores, percentage of A.P. scores that were at or above 31, percentage of 8th grade students meeting state standards on summative tests2, and graduation rates.

Socioeconomic differences between the districts were estimated using the percentage of students in each district who were receiving free or reduced-price meals. Preliminary analysis of the four student academic outcome variables, adjusted for the district’s socioeconomic level, indicated two trends: 1) a linear relationship between lower student academic performance and higher percentages of free and reduced-price lunch students, and 2) “breakthrough” districts with moderate to high rates of poverty but much higher than expected student outcomes.

To select the high-achieving districts used in this report, districts’ performance on the four academic outcomes was compared to the performance levels predicted based on their percentage of free and reduced-price lunch students. The extent to which each district performed higher or lower than expected, in turn, yielded a way to rank districts and to develop a composite ranking across all four measures.

In the fi nal step, to ensure that districts from a range of socioeconomic levels would be able to make meaningful comparisons to the group of high achieving districts, the highest ranking district in the following groups were chosen:

• Districts with less than 25 percent of students receiving free or reduced-price meals (one district identifi ed),

• Districts with 25 to 50 percent of students receiving free or reduced-price meals (largest group, top two districts identifi ed), and

• Districts with more than 50 percent of students receiving free or reduced-price meals (one district identifi ed).

Not surprisingly, some of these districts had already received national attention for their outstanding achievement, which provided some additional face validity for the approach used to select them for this report.

1

A score of 3 is considered by most colleges as the minimum for earning college credit.

2

While immigrant students and those from low-income backgrounds may enter school with language challenges or poor academic skills, this measure indicates how well schools were able to help students overcome these challenges and meet state standards after several years of opportunity.

PII

PII

Appendix C: APQC Process Classifi cation Framework

The APQC Process Classifi cation FrameworkSM (PCF) is a taxonomy of all processes in any organization. The following PCF was slightly adapted to fi t the education industry.

The PCF was created by APQC in 1990 and is being used in thousands of business, government, and health care organizations. APQC is also using this taxonomy to help education improve.

This common taxonomy allows APQC to compare district measures and metrics among these 20 districts and eventually among all districts across the nation in all 12 categories.

The top fi ve boxes are the Categories—the core processes of education

organizations. The bottom seven categories are supporting or enabling processes. Each of the 12 categories is broken down into four to eight Process Groups, which

APQC P

C

F

E

1.0

Develop a Strategic

Plan for the District

6.0 Develop and Manage Human Resources Strategies OPERATING PROCESS

MANAGEMENT PROCESS & SUPPORT SERVICES

2.0 Design, Deliver and Assess Instruction 3.0 Design and Deliver Support Services 4.0 Design and Manage Operations 5.0 Develop and Manage Stakeholder Relations and Services

7.0 Manage Information Technology

8.0 Manage Financial Resources

9.0 Acquire, Construct, and Manage Facilities

10.0 Manage Environmental Health, Safety, and Security

11.0 Manage Intergovernmental and Other Agency Relationships

12.0 Manage Knowledge Improvement and Change > > >

are then broken down further into Processes and then into Activities. Altogether, there are 12 Categories, 72 Process Groups, 314 Processes, and 537 Activities. The PCF category 7.0 Manage Information Technology was chosen by the original 23 pilot districts to be one of the three areas to focus on.

P ROCESS G ROUP D IAGRAM O F 7.0–M ANAGE I NFORMATION T ECHNOLOGY De velop the IT strateg y

Define the enterprise architectur

e

Manage the IT por

tf olio P erf orm IT resear ch and inno vation P erf orm IT

financial management Evaluate and comm

unicate IT

business value and perf

ormance

P

erf

orm IT HR

management Manage IT suppliers and contractors

7.1

Manage the business of information

technology (IT) P ROCESS >>> >>> >>> >>> >>> >>> >>> >>> >>>>>> >>>>>> >>> >>> De velop IT ser vices and solutions strateg y De velop and manage IT ser vice le ve ls P erf orm demand side management (DSM) f or IT ser vices

Manage IT customer satisfaction Mark

et IT

ser

vices and

solutions

7.2

Develop and manage IT customer relationships

De

velop and

manage business reilience De

velop and

manage regulator

y

compliance Perf

orm

integrated risk management De

velop and

implement security

, privacy

,

and data protection contr

ols

7.3

Manage business resiliency and

risk

De

velop the

inf

ormation

management and content management strategies Define the enterprise information architectur

e

Manage information resour

ces

P

erf

orm

enterprise data and content management

7.4 Manage _enterprise information De velop the IT de velopment strateg y P erf orm IT ser vices and solutions lif e cycle planning De velop and maintain IT ser vices and solutions architectur e Cr eate IT ser vices and solutions 7.5 Develop and maintain information technology solutions De velop the IT deplo yment strateg y

Plan and implement changes Plan and manage r

eleases

7.6 _Deploy information technology solutions

De velop the IT ser vices and solution deliv er y strateg y De velop the IT suppor t strateg y

Plan and manage IT infrastructur

e resour ces Manage IT infrastructur e operations Suppor t IT ser vices and solutions Manage IT -specific kno wledge ca ptur e and av ailability Enable collaborativ e w ork 7.7 Deliver and support information technology services 7.8 _Manage IT knowledge

7.1 Manage the business of IT

This process group defi nes and maintains the relevance of IT to the enterprise and its mission and communicates the strategy and role of IT within the enterprise, establishes the enterprise architecture and guiding principles, defi nes the IT

management system and governance model, and manages the strategic activities that help ensure attainment of IT value.

7.2 Develop and manage IT customer relationships

Developing and managing IT customer relationships includes the processes to manage the communications and interactions between the customers of IT and the providers of IT services and solutions; gather business requirements; help the customer to develop justifi cations for requested IT services and solutions; provide the initial interface to IT architects responsible for designing the IT services and solution; and monitors the customers’ satisfaction with the development, deployment, and on going support of the IT services and solutions.

7.3 Manage business resiliency and risk

Managing business resiliency and risk includes the processes required to enable fi rms to rapidly adapt and respond to any internal or external opportunity, demand, disruption, or threat and continue operations without signifi cant impact to the business and enables a more dynamic, strategic, and integrated approach to managing compliance obligations.

7.4 Manage enterprise information

Managing enterprise information is defi ned as understanding what information is important to the enterprise; determining how, when, and from where that

information is collected; defi ning how that information is structured and organized; and establishing the policies and controls needed for the collection and use of that information.

7.5 Develop and maintain information technology solutions

This process group produces, develops, and tests new, enhanced, and improved solutions and enables services that meet all confi rmed requirements; maintains and manages solutions and supports services to continue meeting all confi rmed requirements; and sunset solutions and discontinues services no longer needed.

7.6 Deploy information technology solutions

This process group introduces new, enhanced, or improved functionality into the environment with minimal or no disruption to the services schedules; ensures consideration of appropriate security and contractual constraints; and properly balances business benefi t with technical and business risk.

7.7 Deliver and support information technology services

Delivers IT services and solutions within the budget provided; meets service commitments as measured by customer-defi ned service criteria; achieves customer satisfaction targets.

7.8 Manage IT knowledge

Managing IT knowledge within an enterprise includes managing knowledge of the industry and business of the enterprise served by IT; managing knowledge of the important business processes and business functions supported by the services and solutions developed and maintained by IT; managing knowledge of the enterprise’s information, application, and technology infrastructure architectures and of the services and solutions provided; managing knowledge of processes and techniques for developing, maintaining, and delivering services and solutions, and managing knowledge of the current and emerging technologies underlying the services and solutions provided.

Appendix D: About APQC, OSBC Research, and PIIE

APQC is a nonprofi t organization that is 30 years old. Located in Houston, Texas, at The Houstonian Resort, APQC has approximately 80 full-time employees and a budget of approximately $11 million.

APQC’s mission is three-fold:

• to discover knowledge through research and networking,

• To disseminate knowledge and best practices to help organizations improve, and

• To create new knowledge.

APQC’s board of directors consists of 45 leaders from business, universities, K-12 education, health care, and associations. APQC has approximately 350 members from all sectors.

For more information, visit APQC’s Web site at www.apqc.org/PIIE, call us at 713-681-4020, or fax us at 713-681-8578.

The Open Standards Benchmarking Collaborative

(OSBC) Research

In the early 1990s, APQC received many requests from businesses to help create process performance measures and metrics for their own organizations, and where possible, to compare their benchmark metrics with those of others. Therefore, APQC used the PCF to create a common taxonomy to enable comparisons and collect data from corporations.

In 2004, with the help of many corporations, APQC expanded that work and

created the Open Standards Benchmarking CollaborativeSM (OSBC) research to

systematically collect and store process data not only from U.S. fi rms but also internationally. The OSBC database is still growing.

Open Standards refers to a generic classifi cation of processes across institutions

and sectors. These processes have unique measures where appropriate and are fl exible and changeable as businesses evolve over time. Benchmarking

Collaborative refers to the concept of people and institutions working

collaboratively to build comparable process defi nitions, measures, and metrics for comparison.

Process Improvement and Innovation in Education (PIIE)

The PIIE (Process Improvement and Innovation in Education) project is part of the OSBC research and was created to address the need of K-12 education specifi cally for comparative benchmarking. The PIIE project consists of the following

characteristics.

• Twenty-three districts were selected to participate in a year-long pilot.

• Districts sent representatives to attend a design meeting December 13-14, 2004 and a large kickoff meeting in Houston January 6-7, 2005 with more than 80 attendees.

• Houston session attendees broke into working groups to select key categories and process groups within the PCF to examine.

• Participants selected three areas for the pilot period: instruction, information technology, and human resources (HR).

• Working groups began defi ning measures and metrics for each identifi ed process and sub-process, working collaboratively and for the most part virtually. Core groups were identifi ed to work intensively on the project and report back to the main working groups periodically.

• APQC translated the working and core group results into a Web-based survey for each process area. Surveys are now available in each of the three categories: instruction, information technology, and HR.

• The superintendents of participating districts met for a day and a half (June 17-18, 2005) in Houston to exchange their views of the pilot project, its value to their districts, and their role in using the pilot data.

• APQC blinded the survey results, validated and normalized them, and stored the information in the OSBC database. Then, APQC prepared a report for each district summarizing its data compared to the database.

• Representatives of all districts will come to Houston January 10-13, 2006 for a fi nal meeting a combined one-day knowledge transfer session to share and exchange views and lessons learned as well as create action plans. This meeting will be immediately followed by two and a half days of the Implementation Strategies course, in which APQC conducts train-the-trainer implementation strategies on the use of the data. This combined session concludes the pilot. APQC plans to take the pilot to scale nationwide and internationally over the next fi ve years.

Appendix E: Glossary

Benchmark target

The value of a performance measure in the 80th percentile of the distribution.

Cost effectiveness

Producing good results for the amount of money spent

Cycle time

The length of time it takes to complete a process or sub-process; measured in hours, days, or weeks

Effi ciency

Description of a process that results in less energy being spent to attain the desired product

Full-Time Equivalent (FTE)

To calculate the number of full-time equivalents employed during the year for each respective process or activity, you must prorate the number of employees and the hours spent performing each process/activity. Assume that a full-time worker represents 40 hours per week. Provide the average number of full-time equivalents employed during the year for each respective process. Include full-time employees, part-time employees, and temporary workers hired during peak demand periods. Allocate only the portion of the employee’s time that relates to or supports the activities identifi ed for an applicable process. Prorate management and secretarial time by estimating the level of effort in support of each activity, by process.

For example, a part-time secretary in the fi nance department for XYZ Union School District charges all of his time to fi nance department activities. He works 20 hours per week. The secretary splits his time evenly supporting employees working in the general accounting process and the fi nancial reporting process. Thus, his time should be allocated by process. So, if he works throughout the year and supports these two processes, his time would be split evenly as:

20hrs/40hrs = .5 FTE x 50% for general accounting = .25 FTE for general accounting 20hrs/40hrs = .5 FTE x 50% for fi nancial reporting = .25 FTE for fi nancial reporting

Histogram

A graphic representation of a frequency distribution in which the horizontal axis is determined by the span of values calculated for performance measures and the vertical axis is determined by the frequency that the value occurs (i.e., the number of districts with similar values).

Median

The value at the middle of a distribution; typically, half of the sites for that measure have a value greater than the median and half have a value lower than the median.

Performance measure

A measure of performance based on analysis of standardized data collected from districts using the PIIE surveys; for example, a performance measure on the personnel cost of the recruiting process is computed by taking the total personnel cost of full-time equivalent employees performing that process divided by the total cost associated with the recruiting process; the higher the cost per revenue amount, the less profi cient the process.

Productivity

The measurement of process, labor. or equipment effi ciency when compared to an established baseline

Appendix F: Using This Report

Other factors not contained in this report may infl uence these metrics and outcomes, such as teacher quality, geography, district, state, or school board policies. This was a study of 22 districts as a pilot.

As APQC widens its survey data set and receives more data from additional districts nationwide, we can begin to analyze the effect of many of these different variables on the outcomes.

Nonprofi t APQC has specifi cally prepared this report for your school district based on data provided by your staff. While the data remains completely confi dential, we hope you will share these fi ndings to improve how your district performs routine processes.

For more information, visit APQC’s Web site at www.apqc.org/PIIE, call us at 713-681-4020, or fax us at 713-681-8578.

PII

PII

PIIE

123 North Post Oak Lane, Third Floor Houston, TX 77024-7797

+1-713-681-4020 • 800-776-9676 Web: www.apqc.org/PIIE

E-mail: [email protected]

©2005 APQC. ALL RIGHTS RESERVED.