Calhoun: The NPS Institutional Archive
Reports and Technical Reports Graduate School of Business and Public Policy (NPS-GSBPP)
2008-09-28
Exploring the Link between
Performance and Resource Allocation in
Navy Enterprise
Candreva, Philip J.
http://hdl.handle.net/10945/38087`
Approved for public release, distribution unlimited.
Prepared for: Naval Postgraduate School, Monterey, California 93943
NPS-CDMR-FM-08-002
`ÉåíÉê=Ñçê=ÇÉÑÉåëÉ=ã~å~ÖÉãÉåí=êÉÑçêã
péçåëçêÉÇ=oÉéçêí=pÉêáÉë=
=
Exploring the Link between Performance and Resource Allocation in the Navy Enterprise
25 September 2008 by
Philip J. Candreva, M.S. Natalie Webb, Ph.D.
Graduate School of Business & Public Policy
The research presented in this report was supported by the Center of Defense Management Reform of the Graduate School of Business & Public Policy at the Naval Postgraduate School.
To request Defense Policy and Management Research or to become a research sponsor, please contact:
Center for Defense Management Reform Graduate School of Business & Public Policy Naval Postgraduate School
555 Dyer Road, Room 320 Monterey, CA 93943-5103
Tel: (831) 656-3487 Fax: (831) 656-2253 e-mail:[email protected]
Copies of the Center for Defense Management Reform Research Reports may be printed from our website www.defensereform.org
Executive Summary
OPNAV N09X asked the Center for Defense Management Reform (CDMR) to explore the link between performance and resource allocation in the Navy
enterprise. This is a report of a study conducted of the Surface Warfare Enterprise (SWE) in the summer of 2008 to assess how resource allocation decisions are made, how readiness (performance) is measured and reported, how well the SWE is linking the two, and to identify hurdles and enablers for success.
Performance-based management systems (PBMS) provide a framework to help organizations achieve desired outcomes by measuring and evaluating its
efficiency and effectiveness. We looked at the SWE through a PBMS framework and provide five findings with associated recommendations:
x The SWE understands there are both political and economic elements to performance management in the public sector and is taking appropriate steps to manage both. To be most successful, political influence strategies should employ tactics and information beyond what the enterprise’s own performance management system generates.
x There is commendable discipline in data collection and documentation processes, which should continue. Communicating performance should be more precise, however. Specifically, the concepts of efficiency and cost-effectiveness are co-mingled when they should be distinct.
x The enterprise has developed many measures of readiness and proxies for readiness. Attention should now shift to evaluating the efficacy of those measures. It is likely there is more data captured and displayed than is useful and the enterprise should limit the number of measures to those that convey the most information, that is, those that are the most useful for decision-making.
x Data quality plagues most large, complex organizations and the Navy is no different. We recommend the measures used by decision-makers
acknowledge the degree of uncertainty in the data. We recommend the enterprise continue efforts to clean databases. We further recommend Navy Enterprise link their performance measurement efforts with the Navy’s financial management improvement efforts to construct cost accounting capabilities that will augment existing budgetary accounting systems.
x Since most public sector outcomes are subjective, we recommend the enterprise consider the use of more subjective measures, particularly when linking ouputs to outcomes and to support their political influence strategies.
Mapping objective measures to subjective measures in a manner that shows causal relationships will keep the effort focused and will support budgetary requests.
In summary, the Surface Warfare Enterprise has laid a solid foundation for their performance-based management system. They have an accurate self-assessment of many of its strengths and weaknesses. We encourage them to continue on the path they are on with attention shifting in the few key areas noted above. Within the report are specific examples of SWE products, examples of
successful performance management systems from other government entities which may inspire new ideas, and references to sources of additional information.
About the Authors
Phil Candreva’s research investigates how government organizations use financial information in such areas as resource allocation decision-making,
accounting, performance measurement, and management reform. Most
contemporary public sector management reform efforts are either explicitly tied to financial decisions (e.g., performance-based budgeting) or are implicitly tied through other management efforts (e.g., efficiency programs). Since budgets are the
battlefield on which public policy disputes are waged, public managers must become proficient at showing how effectively and efficiently those resources are being used in order to preserve or expand their resource base. Such efforts are a critical dimension of contemporary management reform.
Phil Candreva Senior Lecturer
Graduate School of Business and Public Policy Naval Postgraduate School
Monterey, CA 93943-2884 Tel: (831) 656-3622 E-mail:[email protected]
Natalie Webb’s research explores multiple areas of performance management, to include strategic planning, implementation of strategic plans, performance management systems, measuring performance, and performance evaluation of executives. She also studies the interaction among defense and non-governmental organizations (NGOs) as they coordinate and combine action among diverse actors to provide security, stability, and in some instances, reconstruction after war or natural disasters. Her research and teaching, based on theory, historical analysis and current practice, contributes to better management practices both inside and between varying types of organizations
Natalie Webb Associate Professor
Graduate School of Business and Public Policy Naval Postgraduate School
Monterey, CA 93943-2013 Tel: (831) 656-3622 E-mail:[email protected]
Disclaimer: The views represented in this report are those of the author and do not reflect the official policy position of the Navy, the Department of Defense, or the Federal Government.
NPS-CDMR--FM-08-004
`bkqbo=clo=abcbkpb=j^k^dbjbkq=obcloj
péçåëçêÉÇ=oÉéçêí=pÉêáÉë=
=
Exploring the Link between Performance and Resource Allocation in the Navy Enterprise
25 September 2008 by
Philip J. Candreva, M.S. Natalie Webb, Ph.D.
Graduate School of Business & Public Policy
Table of Contents
Introduction ... 1
Background ... 1
Methods ... 3
Benefits of the Study ... 4
Limitations of the Study ... 4
Structure of the Report ... 5
Performance-based Management Systems ... 7
Overview ... 7
Critical Terminology ... 7
A Framework for Performance-based Management ... 11
Impediments to successful performance management ... 15
Conclusion ... 18
Findings and Recommendations... 19
Finding 1: Balance political and rational means to your ends... 19
Finding 2: Extend process discipline to communications ... 23
Finding 3: Evaluate the effectiveness of the measures ... 25
Finding 5: Consider employing forward-looking and subjective
indicators... 37
Appendix 1: Summary of Recommendations... 45
Appendix 2: Additional Information ... 47
Books and Articles: ... 47
Web-based resources ... 48
Introduction
OPNAV N09X asked the Center for Defense Management Reform (CDMR) to explore the link between performance and resource allocation in the Navy
enterprise. This is a report of a study conducted of the Surface Warfare Enterprise (SWE) in the summer of 2008 to assess how resource allocation decisions are made, how readiness (performance) is measured and reported, how well the SWE is linking the two, and to identify hurdles and enablers for success.
Background
Since at least the Hoover Commission in 1947, the federal government has sought better ways to measure its performance and to allocate resources in a manner that increases the overall performance of government. Over the past 60 years, government organizations have used various performance-based
management systems (PBMS) to attempt to link resources with results.1
In one way or another, nearly all PBMS stress a set of measures and goals, based on a logical flow from strategy through metrics to evaluation (and back through in the opposite direction), that ultimately focuses on results or outcomes, and how to achieve those results or outcomes in a cost-effective manner. PBMS is the subject of the next section of the report.
In practice, government organizations have found it extremely difficult to connect outputs and outcomes to resources. With each new PBMS (or
transformation), the federal government attempted to use a performance basis for allocating resources across agencies. Agencies then sought performance bases for
1
The efforts by Navy Enterprise follow in a long path of attempts to link performance to budgets. Past attempts, which have had varying degrees of successes, include planning-programming-budgeting (PPBS) in the 1960s, management by objectives (MBO) in the early 1970s, zero-based budgeting (ZBB) in the late 1970s and 1980s, GPRA and the performance movement of the 1990s and its evolved form, the President’s Management Agenda and PART rating tool of the 2000s.
allocating across functions or across departments within agencies. With enterprise-wide information systems and standard, repeatable business processes, agencies increased their ability to measure consistently. Transformational change through better understanding of the agency, however, does not come simply from faster and more reliable data. Timely and reliable data exist only to support improved decision-making; thus, data need to be linked to improved processes. Change needs to occur in the business processes and the culture of the organization in which those
processes occur and that those data describe. Navy Enterprise is an attempt to make such transformational change. From the Navy Enterprise website:
“Navy Enterprise is about improving the return on investment in all that we do. It is about managing initiatives to guarantee the appropriate balance between efficiency and risk. It is about becoming more innovative in our strategy and processes to counter balance the growing financial challenges we face. Navy Enterprise is about
maximizing our resources and continuing to effectively deliver combat ready maritime forces to joint commanders now and in the future—it's about achieving the objectives identified in our new Maritime
Strategy.”
The Navy has made critical strides towards collecting consistent and timely data. The next steps will be to ensure the data can be analyzed and used to manage resources more efficiently and effectively. Accordingly, we sought to (a) understand the existing relationship between organizational performance and resource allocation within a subset of Navy Enterprise and (b) provide
recommendations based on studies of performance management and performance-based budgeting in the academic and practitioner literature. Specific research questions included:
How do managers make resource allocation decisions?
How is readiness measured and reported?
How is readiness reporting linked to financial resources?
What lessons can be taken from studies of successful performance based management systems?
Methods
We employed a case-based research design, informed by scholarly and practitioner literature on performance-based management. Case-based research is common in public administration, the social sciences, and business research and is particularly useful when:
the research questions are in the form of “why” or “how”
the research subject is a complex phenomenon
the phenomenon is unique and contemporary
the study does not require experimental control.2
All factors were applicable to this study. The Surface Warfare Enterprise (hereafter, SWE) was chosen due to its relative maturity among the warfare enterprises. It has been working this question for a while, but has not
institutionalized its practices yet. The potential to use the findings appeared greatest.
We reviewed scores of briefings, documents, meeting minutes and other archival information on the SWE Intranet. We conducted about twenty-five hours of interviews with members of the enterprise, representing cross-functional teams, class squadrons, and contractor support. We reviewed academic and practitioner studies of performance-based management systems. And we reviewed promising practices3 in other government activities whose experience may benefit Navy
Enterprise.
2
Robert K. Yin. (2003) Case Study Research: Design and Methods, 3rd ed. (Thousand Oaks, CA: Sage).
3
Many would have used the term “best practices” here, but since the concept of “best” is context-specific, best practices cannot be exported to another context and still be expected to be best. Promising practices is more descriptive.
We did not attempt to evaluate the readiness of the surface force, nor did we attempt to evaluate the effectiveness of the use of financial resources. We also did not evaluate whether there were enough resources. Rather, we evaluated the processes used to measure performance and the processes used to allocate resources with an aim to improve those processes.
Benefits of the Study
The primary benefit of this study is educational. The Naval Enterprise initiative is vast and encompasses virtually all of the Navy’s organizations and processes so successful practices in one domain may not be made available to others. Knowledge management is difficult in large organizations. The primary benefit of this study is the combination of lessons from evaluating current practices inside the Navy, evaluating practices outside the DoD, and the ultimate transfer of that information through the enterprise. This report, then, is structured to be more informational than evaluative.
In addition to this report, we are preparing a teaching case study that will be available for use in the Naval Postgraduate School’s defense-focused MBA
program, the Navy’s Center for Executive Education, and the Defense Resources Management Institute. The findings and recommendations in this report will be used to educate future naval leaders.
Limitations of the Study
We covered only one warfare enterprise so the findings may not be fully applicable to the other warfare enterprises. Applicability is a function of the other enterprises’ approaches to performance management and the level of maturity of their processes. We also emphasized current operations and current readiness. More work was done in this domain by the SWE than expected, and that limited the researchers’ time to explore the SWE’s role in modernization or personnel
the readiness data collection and reporting resulted in a bias toward the analysis of operations and maintenance (O&M) spending. The SWE financial footprint is significantly larger, especially when one considers the surface warfare portion of provider enterprise activities (viz., manpower and acquisition spending). Future studies should consider these larger accounts to get a more complete picture of the financial resources devoted to surface warfare readiness.
Structure of the Report
This report is arranged in two major sections. The first describes generally the concept of performance-based management systems and applies that concept to the SWE. This foundation is necessary to establish as the findings and
recommendations are grounded in that framework. The second section is organized around six major findings; each finding is described, as are its consequence or importance, our recommendations for improvement, and additional information the enterprise may find helpful. Two appendixes summarize those recommendations and sources of information.
Performance-based Management Systems
Overview
Performance management, by one definition, is the systematic process by which an agency involves its employees in improving organizational effectiveness in the accomplishment of agency mission and goals. Rather than focusing on inputs and work being done, which is what government agencies have traditionally done, PBMS provide a framework to help an organization achieve desired outcomes and push managers to measure and examine results. Recent research summarizes the set of performance management challenges faced by any government manager: to improve effectiveness, focusing on how well desired outcomes are achieved; to improve efficiency, focusing on how well the costs of producing goods and services are managed and, to improve accountability, focusing on bringing together budgets and performance measures. 4 In addition, public sector managers use performance management to help them tackle a set of specific managerial issues including evaluation and feedback.5
In this section we define inputs, outputs and other terms needed to discuss a government organization’s ability to measure and evaluate performance.
Critical Terminology
At the heart of PBMS lie two essential concepts: effectiveness and efficiency. To correctly measure performance and fund in line with priorities requires
4
Melese, F., Blandin, J., & O'Keefe, S. (2004) “A New Management Model for Government: Integrating Activity Based Costing (ABC), the Balanced Scorecard (BSC), and Total Quality
Management (TQM) with the Planning, Programming and Budgeting System (PPBS),” International Public Management Review, 5(2): 103-131.
5
See, for example, Robert D. Behn (2003) “Why Measure Performance? Different Purposes Require Different Measures,” Public Administration Review 63(5): 586-606.; Steven J. Kelman (2006) “Public Management Needs Help!” International Public Management Review. 7(1): 1-5.; and Theodore H. Poister (2003) Measuring Performance in Public and Nonprofit Organizations (San Francisco: John Wiley & Sons, Inc).
understanding the connections among an organization’s goals and its resources allocation processes. Desirable metrics flow from the goals and show whether a program or organization is effective and efficient. Measuring efficiency and effectiveness, however, requires bridging two separate perspectives of an organization.
On the one hand, managers want to know whether an organization or program is effective. Effectiveness measures describe how well a program or organization achieves a strategic goal; they are the outcomes – the results or consequences of direct importance to program customers or the public. For example, “customer satisfaction,” “winning the war” or “being well-educated” are outcomes. Effectiveness measures come from the right-hand side of the model in Figure 1:
Figure 1: Resource Allocation to Outcomes
On the other hand, budgeting processes by their nature focus on inputs (line items and appropriations). Organizations use their funding to buy inputs (resources used, e.g., labor, equipment, etc.) to produce an output, which we define as the completed product of internal activity or work done within the organization or by its contractors (hours steamed, flight hours, numbers of jobs done by Seabees, etc.). To manage well requires understanding the outputs and being able to measure them. We ask organizations to be efficient – to use the fewest inputs at the minimum cost necessary to produce a certain amount of output. Measures of efficiency describe how well an organization uses inputs to produce outputs. In contrast to the examples of outcomes from the previous paragraph (“customer satisfaction,” “winning the war” or “being well-educated”), corresponding output measures might be providing meals, readiness, or classes that, when properly
provided (in a timely manner, of a certain quality, meeting certain standards, etc.), can result in desired outcomes.6
Outputs and outcomes can be difficult to identify and assess. Rather than being natural measures (those with common interpretation, such as dollars to measure cost), we often use proxy measures. Proxies do not directly measure performance, but are related to what we want to assess. Consider the issue of public safety. Typical measures used to assess public safety include rates for murder, rape, robbery, and the like. These, however, are outputs – they serve as proxies for the desirable outcome of “a safe neighborhood” or “a safe city.” Even assessing how members of the public feel about their safety, an outcome measure, can be problematic; while this is the outcome desired by many, if crimes are not reported to the public or are sensationalized in the media, accuracy of responses to surveys will not reflect whether safety is “better” or “worse” over some time period.
In some instances, the difficulty in measuring performance forces us to assign or construct measures to describe different levels of performance. For example, when assessing activity limitations in daily living, the National Health Interview Survey assesses a long-term reduction in a person’s capacity to perform the average kind or amount of activities associated with his or her age group. People are classified into one of four categories:
1. unable to perform the major activity,
2. able to perform the major activity but limited in the kind or amount of this activity,
3. not limited in the major activity but limited in the kind or amount of other activities, and
6
Outcomes definitions adapted from Harry P. Hatry (2001) “What performance measures should be tracked?” in Dall W. Forsythe, ed Quicker, Better, Cheaper: Managing Performance in American Government (New York: Rockefeller Institute), p. 18.
4. not limited in any way.
Measures such as these have direct applications in defense, where outcomes often have no natural or easily determined proxy measure.
In addition, one’s definition or view of performance depends on one’s
responsibilities, or “where you sit.” For example, a rescue mission may have more than one end outcome; a rescue may save a life, save equipment, and constitute a “win” for an organization. In the process of preparing for and attaining the outcomes, the organization may measure other things thought to increase the probability of success in performing rescues. For example, the organization may perform drills, measuring the time it takes to instigate the rescue process. It may perform safety and other quality repairs and drills to be prepared to undertake rescues. These actions, whether defined as outputs or intermediate outcomes, are thought to be predecessors to success in achieving the organization’s higher-level goals.
In summary, care must be taken both in the definition and construction of metrics to assess output and outcome. Outputs reflect the product of work or activity and outcomes are results, accomplishments, effects or impacts. When applied to government situations, it often becomes necessary to use outputs as proxies for desired outcomes, or to construct measures of performance related to the outcome desired.
Performance Based Management Framework $$_____ $$_____ $$_____ $$_____ $$_____ $$_____ $$_____ Budget Authority
Activities
The rational world of efficiency
The political world of public choice The less rational world of
budgeting The uncertain world of effectiveness
Environ-mental factors Environ-mental factors Environ-mental factors Environ-mental factors Environ-mental factors measures measures measures
Figure 2: Performance Based Management Framework A Framework for Performance-based Management
Using the terms we have defined, we now move to Figure 2, which illustrates the framework we used to examine SWE performance. At the center of Figure 2, inputs are transformed by activities into outputs. This is what we call the “rational world of efficiency,” and results in measures of efficiency useful for internal planning, motivation and control.
The framework also shows the differences among the terms budgeting,
inputs, outputs and efficiency: without a direct connection from expenditure on inputs to outputs produced, efficiency cannot be measured. Additionally, in the “less
rational world of budgeting,” many factors related to budget authority affect the use of inputs in productive processes; these may or may not improve efficiency.
Outcomes, or those results, effects, or accomplishments that happen when the organization uses an output towards achievement of a goal, depend on external factors and human preferences and behavior. We refer to this as the “uncertain world of effectiveness.”
Finally, environmental factors outside control of the organization’s leadership affect all stages of organizational performance, including whether this particular value stream is the preferred means to attain the desired outcome. We call this the “political world of public choice.” A specific example shown as Figure 3 can help clarify:
$150 Gas $30 Oil $300 Tires $100 Labor
Performance Based Management Framework
How can I increase fuel efficiency?
What if I took the bus or a taxi instead? How much should I budget for
gas? How much do my car and my driving habits affect this?
Inflation Road conditions Manufacturer recommend-ations Traffic Congestion Temporary work location Commuting time M.P.G.
Cost per fill-up
Driving & maintaining the car Passenger - miles driven # Trips taken Punctual arrival at designated place
In this example, assume transportation will be by private car. The budget includes money to buy inputs such as gasoline, oil, tires and labor expenses. Suppose the car gets 25 miles per gallon (mpg), our monthly gasoline budget is $150 and gasoline costs $3 per gallon. All other things held constant, we can buy 50 gallons of gasoline and drive 1250 miles in the month. Fuel efficiency of the car is 25 mpg, and the more typically used unit cost measure of efficiency is
$150/1250=$0.12 per mile. Miles driven represents the output (although other measures, such as trips taken, are possible).
In the rational world of efficiency, the question is, “How can I increase fuel efficiency?” Perhaps filling the tires with more air will help. Perhaps changing the oil or getting a tune-up will help. Driving a steady speed and accelerating gradually also improve fuel efficiency. This question involves process improvement, where changes in the mix of inputs, increases in technology and training, and other ways to “lean” the process result in improvement.
In the less rational world of budgeting, we face the more difficult question of “How much should I budget for gas?” If gasoline goes to $4 per gallon, with the same budget we buy 37.5 gallons and drive 937.5 miles. Fuel efficiency is still 25 mpg, but the unit cost measure of efficiency changes to $150/937.5=$0.16. Another possibility is to fund to the same workload, costing $200. Fuel efficiency remains at 25 mpg and unit cost efficiency again is $200/1250=$0.16. For next month, should we budget $0.12, $0.16 or another figure per mile driven? As one might suspect, prices of inputs and the amount of the output affect the answer. How much work we plan to do, and at what prices, should help determine the amount of budget for each input, NOT the amount spent this year. These measures also highlight the difficulty in measuring efficiency.
Note that this relationship separates budget from output: increasing the
the outcome. Similarly, it is probably impossible to derive a unit-of-readiness-per-dollar measure that is accurate and appropriate for budgeting.
Outcome depends not only on how many miles we can drive, but whether we get where we want to go in a timely manner. Many other factors affect the outcome including the age of the car, our schedule, the need to have flexibility to come and go as needed, traffic, weather, and road conditions, and other factors out of the driver’s control. It is also situational: deciding whether to drive back and forth to work in Monterey, California is quite different from deciding whether to drive in Bangkok or Mexico City, where two-hours traffic jams occur on a regular basis, and driving to work in a metropolitan area when one works third shift may be quite different than driving for a 9 to 5 job.
Figure 4 shows the same relationships as the car example using possible concepts and measures from the SWE.
SWE in the Performance Based Management Framework $$ O&M,N $$ MPN $$ RDTEN $$ SCN $$ OPN $$ WPN Budget Authority
Activities
The world of MFOM and TFOM
Other services, or Navy communities competing for missions The amount obligated or
available to buy resources
Do the outputs generate a ready ship?
Performance Based Management Framework
Environ-mental factors Environ-mental factors Environ-mental factors Environ-mental factors Environ-mental factors
Figure 4: SWE Performance Based Management Framework
Impediments to successful performance management
We would be remiss if we did not point out a few terms that should be used extremely carefully or should not be used at all in reference to performance
measurement (due to the likelihood that they misconstrue proper indicators of performance) and a few concepts that should guide both our choice of performance management systems and measures and the way we communicate them.
Terms
The term, “cost,” a sacrifice of resources, may be an outlay cost (past, present or future cash flow) or an opportunity cost (foregone alternative benefit). Cost, expenditure and inputs often refer to different things. Strictly speaking, only the cost of inputs used in production of the output should be considered in measuring efficiency. Additionally,
without robust accounting systems (such as activity-based costing) to tie input costs to specific outputs, we cannot precisely measure efficiency. Without connecting input costs to outputs, we have no possibility, except at the most rudimentary, macro level, to connect costs to outcomes.
The overused term return on investment (ROI) applies to investments, not expenses. In DOD, ROI should help guide investment decisions (e.g., buying a new ship), but is not an appropriate measure to help manage operating expenses (e.g., operating the ship we already have).
Concepts
Some basic concepts cover the key points in choosing useful performance measures. First is to focus on those measures connected to strategic goals that also show a clear relationship between the outcome and the measure. Where possible, measures should be direct; for example, in measuring job safety, percent
compliance with safety guidelines is an indirect measure, while number of worker injuries is direct. Measures should be comprehensive, and operational. If values are not relatively easy to measure, practical to implement or observe, and relatively easy to interpret, they will not be used to drive action towards the achievement of the most important strategic goals.
“Metric mania,” a condition in which the sheer number and disorganization of metrics makes evaluating, comprehension and accountability problematic, should be avoided.7 In most organizations, many things are measured that have no direct tie to the success of the organization. Inputs, activities, tasks, and work may be important to measure if they contribute to achieving results; however, by
themselves, they provide little to no value.8 An organization must focus on measuring output and outcomes to effectively manage performance and meet the organization’s goals.9
The concept of sub-optimal behavior applies to performance
7
See, for example, W. Casey, W. Peck, N.J. Webb, & P. Quast, (In press) “Are we driving strategic results or metric mania?” International Public Management Review.
8
management when it is done at multiple levels of an
organization. Using performance-based measures can result in sub-optimizing behavior if goals do not support higher-tier goals. Without a broader view, each smaller organization can perform effectively and efficiently, but may not be effective or efficient from the perspective of the Navy as a whole if there is
redundancy in activities or organizations focused on lower-tier goals that do not contribute to goals of the larger enterprise. Finally, some of the difficulties in implementing performance
management in government stem from several basic
differences between private and public organizations. Two of these are the ability to measure the product or output of work, and the control managers have over inputs into the processes needed to create outputs and results. Figure 5 shows the relationship among private, nonprofit, and government organizations in working with inputs and outputs.10
As this figure shows, government organizations tend to have the least ability to measure what they do, and less direct control over inputs used to do the work. Because of this, the quality and efficacy of the performance management system cannot be as robust – simply put, the Navy cannot do performance management as well as Toyota.
10
Peter Frumpkin & Joseph Galaskiewicz (2004) “Institutional Isomorphism and Public Sector Organizations,” Journal of Public Administration Research and Theory, 14(3):283-307.
Figure 5: Relationships among private, nonprofit and government organizations in the ability to measure outputs and control inputs Conclusion
What, then, can an organization do to begin to tie performance to budgets? First, it should provide a clear understanding of what it means to be efficient and effective, and an understanding of the cause and effect relationships modeled in Figure 1. Second, the organization can help their personnel and other stakeholders understand the factors affecting program or organizational performance. And not all the factors are related to resources: it is necessary to be absolutely clear about what can be “controlled” by resources and what cannot. In the absence of performance measurement and accounting systems that connect performance and levels of funding, managers can concentrate only where activities, outputs, and outcomes are reasonably associated with and responsive to funding levels. An analysis of
performance and resource allocation activities in one organization can shed insights on how and where to link budgets to performance. At the end of the day, of course,
we doing things right?” (efficiency) should drive all discussion of performance management and measurement.
Findings and Recommendations
Our evaluation of the SWE, examined through the lens of performance-based management, resulted in five findings. Each finding has several recommendations for improvement. A finding should not necessarily be considered a negative thing; some findings are quite positive where others are neutral. All are presented in a manner we believe will add value to the SWE and the broader naval enterprise.
The findings come from the fields of economics, knowledge management or decision science. In them, we address the quality of data, the effectiveness of the measures the data support, the breadth and type of measures, the way they are communicated, and the decision-making processes they support.
Finding 1: Balance political and rational means to your ends Description
Performance-based management systems must deal with two fundamental types of issues, the rational economic ones and the less rational political ones. The SWE is taking appropriate steps to manage both through its metrics efforts and its influence strategies. They have mapped all the financial resources that directly or indirectly support their desired outcomes and have identified the organizational entities responsible for those resources. See Figure 6. Because the SWE chief executive does not directly control most of these resources, the enterprise is developing an influence strategy.
The strategy recognizes the political basis of decision-making. Political, in this context, is not derogatory; it simply refers to any group decision-making process in which the members of the group have disparate goals. Such decision-making is
based partly in power and partly in coalitions.11
The SWE is basing its strategy on the critical interorganizational relationships and the processes embedded in the Navy’s resource allocation system. The newly established CFO, staffed with an SES, is an appropriately placed rank to fulfill the role of exerting influence on the supporting organizations and processes.
Figure 6: SWE Financial Resources Consequence
There has long been a desire among government managers and budgeters to find a rational basis for the allocation of resources. Going back nearly 70 years, V.O. Key raised the question, “On what basis shall it be decided to allocate x dollars to
activity A instead of activity B?”12
While a microeconomist would say one should put the next marginal dollar on the activity that generates the most marginal utility, doing so in practice is difficult. Key noted that, “The doctrine of marginal utility, developed most finely in the analysis of the market economy, has a ring of unreality when applied to public expenditures.” In the end, Key acknowledged it is a fundamentally political decision. Public budgeting research has since failed to prove Key wrong.
In fact, recent research shows that budget allocation decisions by
legislatures, for example, are not influenced by agency-generated performance information.13 However, research also shows that performance information is very
useful for internal agency management.14 Insofar as the organization finds the
performance information useful for day-to-day management of the activities, it is valuable and should continue. The organization should also understand that the political decision-making process discounts such information and relies on other factors. Thus, it is appropriate for the warfare enterprises to gather performance information in their role as managers and stewards of military capabilities; they are cautioned, however, not to expect that same information to drive resource allocation decisions outside their sphere of direct control. Tools of political influence are most effective for those matters.
Recommendations
Continue to apply political solutions to the political world and apply economic ones to the rational world. That is, in the PBMS framework, focus your influence
12
Key, V. O. (1940). The Lack of a Budgetary Theory. The American Political Science Review, 34 (6): 1137-1144
13
Carolyn Bordeaux (2008) “Integrating Performance Information into Legislative Budget Processes,”
Public Performance & Management Review 31(4): 547-569.
14
See, for example, Gerald Miller, W. Bartley Hildreth, and Jack Rabin, eds. (2001) Performance-based Budgeting (Boulder, CO: Westview Press) or Carolyn J. Heinrich & Laurence E. Lynn, Jr. (2000) Governance and Performance: New Perspectives (Washington, DC: Georgetown University Press).
strategies on the budgeting portion (linking dollars to inputs) and the overall choice (linking SWE-associated dollars to desired outcomes). Influence strategies should certainly include bringing to bear the evidence from performance measurement, but augment that line of reasoning with other tactics. Budgeting decisions are influenced more by external validations of performance than by a program’s self-reported
claims, so use external evidence when available. Information generated by a decision-maker’s staff is more influential than information generated externally, so employ that information when available. Most importantly, the decision-maker’s personal experience affects budgeting decisions, so influence strategies should be personalized.
While influence strategies contend with the political and less rational
relationships in the PBMS, the enterprise should focus its measurement, evaluation, and efficiency-seeking efforts in the more rational relationships between inputs, activities, and outputs. This is where efficiency-seeking initiatives (e.g, Lean Six Sigma proposals) will have the greatest and most certain effect. Given the uncertainty of the political aspects of resource allocation decision-making, the rational processes should make allowance for that uncertainty.
Additional Information
The External Control of Organizations: A Resource Dependence Perspective by J. Pfeffer & G. R. Salancik
Creating Public Value by Mark Moore
Performance Budgeting: Efforts to Restructure Budgets to Better Align Resources with Performance GAO Staff Study (GAO-05-117SP), February 2005
Linking Performance and Budgeting: Opportunities in the Federal Budget Process by Philip Joyce, IBM Center for the Business of Government, October 2003.
Finding 2: Extend process discipline to communications Description.
There is commendable discipline in certain processes, particularly in data collection and documentation. There is a formalized process for documenting the bases for measurement including definitions, sources, manipulations, presentations, and responsible individuals. This practice, while administratively burdensome, is essential to a repeatable, verifiable process. Given the rate of personnel turnover, this practice will sustain reform momentum.
In other areas – particularly in communicating performance – there is less discipline. The most notable is that cost and spending are often used
interchangeably yet there are significant differences between these concepts. Figure 7, for example, is taken from an enterprise presentation.
Figure 7: Communication Example
The top of the figure shows the basic organizational process of converting inputs to outputs through some transformational activity, but the chart, which is intended to display the concept of productivity, falls short of its goal. The vertical axis is labeled, “Input: $ Spending” inferring this is the budget authority used and not the inputs acquired with that budget authority. The horizontal axis is labeled,
cost control. Having a dollar sign on the vertical axis and cost on the horizontal could lead one to believe this depicts changing prices of inputs. To illustrate productivity, a better chart would eliminate the qualifiers on both axis labels and simply plot inputs against outputs.
The curve on the slide is notional, but implies a relationship. Unless one actually evaluates a given input:output pair, one does not know whether the line is concave, convex, or a step function. It may be bounded, or it might even be
horizontal or vertical if there is not a clear relationship. Presuming a particular shape could lead to suboptimal decisions.
The text to the right is also misleading. Management may have no control over the price of inputs and thus does not have the power to move the organization as indicated. (Recall the discussion of the automobile in the previous section.) They can affect the ratio of inputs to outputs through efficiency efforts, but not necessarily dollars to outputs. For example, you may be able to reduce the labor hours to
perform a task, but you have no control over the pay raise that Congress will enact.
Consequence
By focusing on the amount of spending relative to a unit of output, the analysis is confounded by both price effects and program effects. The enterprise managers have control over the program effects, but often not the price effects. Even when they do have control over the price effects (negotiating terms of a contract, for example), the contracting process is still a separate activity from the process that converts the input to an output.
The issue is most pronounced at the fiscal year transition. To illustrate the consequence of miscommunication, let us assume that the enterprise receives windfall funding late in the fiscal year. The funds are obligated on spare parts to fill storerooms on ships. That practice is sound, but that spending is not a cost to the
start of the second year. The organization simply converted one asset (budget authority) into another asset (repair parts). It is not until the asset is consumed in some activity (maintenance) that a cost is incurred.
Recommendations
Focus efficiency efforts on the relationship between inputs consumed and outputs generated, not on the dollars spent. This means tracking labor hours instead of labor dollars. It means tracking parts consumed, not the obligation of funds for parts. Productivity or efficiency is in that relationship. Linking the acquisition or use of inputs to the funds obligated for them is a different analysis, a cash flow analysis; it is not an efficiency concern. The discussion in finding #4 on cost accounting is related.
When communicating the success or challenges of the enterprise in a budgeting environment, be certain to separate the two effects: price concerns and programmatic needs.
Additional Information
“Linking Performance to Funding Decisions: What is the Budgeter’s Role?” by Gloria Grizzle, appears in Public Productivity and
Management Review.
Performance-based Budgeting by Gerald Miller, W. Bartley Hildreth, and Jack Rabin
Visual and Statistical Thinking: Displays of Evidence for Decision Making by Edward Tufte
Finding 3: Evaluate the effectiveness of the measures Description.
While the data are well defined, the role of data in SWE decision support is not as well defined. Finding 2 noted the quality of documentation and within the basis for measurement the user is prompted: “How will the metric be used to change
behavior?” and “How does the chart contribute to improving warship readiness?” Too often, however, these were answered with general statements about leadership probing the issue and taking action. Specific actions, and the triggers that require such action, are not specified.
In other cases data were arranged in ways that provided ambiguous information that is not effective for decision-making. The field of knowledge
management tells us that data, such as a specific measurement of a component of readiness or the status of funds, become information in a given context. Information, in turn, provides meaning and context for action. Knowledge enables that action. A knowledgeable person can make good decisions and act only in the presence of appropriate information that is based on appropriate data.15 It is important that the
enterprise convert data into meaningful management information. We illustrate with an example; see Figure 8.
Figure 8: Sample Bridgeplot
This chart displays a measure of ships’ material condition against spending. Looking more closely, we see the bars represent the number of ships whose material condition (MFOM = maintenance figure of merit) is such that they are deemed “ready” or “not ready.” A small number of ships had no data available and other ships were in shipyard availability periods and so MFOM is not currently meaningful. The bars are not point descriptions, but rather 12 month rolling
averages. The lines on the chart represent planned obligations of budget authority (triangles) and actual cumulative obligations (squares). At the transition of fiscal year, the lines were connected so that the portion of the line prior to October 2007 represented fiscal year 2007 funds and the portion after represent fiscal year 2008 funds.
What information does this chart convey? Looking at the stacked bars, one would conclude that the number of ready ships is down slightly over the four-month
period (presumably not a good thing), the number of ships with no data is nearly zero (presumably a good thing), and the number of ships in availability has
increased (unknown whether this is good or not). As a 12-month rolling average, one does not know the current status of the fleet, just the trend over the last year.
Looking at the lines, one sees the planned spending line is a step function at fiscal quarter intervals, inferring this is actually depicting allocated funding and not planned obligations. One also sees that the amount of funds dropped to zero and began climbing again. This, of course, is due to the change in fiscal year, but by linking the FY07 line to the FY08 line, it looks like a single account, not two different accounts. Since the bars and lines are portrayed together, there is a presumed relationship, but the chart depicts a rolling average of maintenance (an evaluation measure) against current status of funds (a control measure). It is unclear from the chart what decisions this should drive.
Consider another example; see Figure 9. The enterprise is responsible for the readiness of its warships and readiness is said to have several components. The SWE defines readiness as a function of personnel, equipment, supply, training and ordnance, commonly referred to as the PESTO pillars. This is consistent with the logic underlying the Defense Readiness Reporting System (DRRS). Data are collected along these five pillars. Cross-functional teams tend to concentrate on the health of the fleet within one or more of those pillars. The class squadrons look at the health of the ships across the pillars. Figure 9 is a chart used by one of the class squadrons to assess the effect of staffing on ship readiness. We see the chart is arranged by the DRRS structure of missions the ships may be tasked to perform. Under each mission area are training figures of merit (TFOM) and manning data.
There are some concerns with Figure 9. First, the second column from the left after the unit identification is all red. There are three other columns that are nearly all red. If an indicator is red for all units one should question whether the stoplight scheme is miscalibrated since it is unlikely that all ships are not ready. The tolerance for the red designation may need to be adjusted. There is another danger with stoplight coding of measures. Depending how that measure is used, there are incentives for suboptimal allocation of resources. Taking a simple example, if there are two areas of deficiency, one scored at 82 and the other at 89 and both are coded blue, there is an incentive to correct the 89 deficiency before the worse 82 deficiency because the 89 will improve its characterization (turn green) with a one point gain; the 82 requires an eight point gain. The color scheme perversely leads one to correct the better area before the worse area.
It should be mentioned that during this study two surface ships scored poorly on their inspections by the Board of Inspections and Surveys (INSURV). The SWE embarked on a “redline” effort to understand how well their performance
measurement system does or does not predict or record poor performance. Such an effort is exactly the type of evaluation we recommend, below. While that worthwhile
effort was prompted by an unfortunate event, it is characteristic of the type of evaluation of measures that should be done routinely.
Consequence
Generally speaking, there are several reasons why performance should be measured. Performance may be measured to evaluate how well the organization is doing, to control critical processes, to base resource allocation decisions, to learn what is working and what is not, to motivate managers and employees, or to promote the success and value of the organization. (Many of those are actually intermediate purposes for measurement; the ultimate goal being improved
performance.) The selection and presentation of measures should be based upon one of those reasons: evaluation, control, allocation, learning, motivation, or promotion.16
No one set of measures will serve all purposes. Managers should look at their specific goals, responsibilities, and decisions and select that set of measures that best supports their work. For purposes of evaluation, measures should assess outcomes in manner that is comparable against some standard (e.g., manning vs allowance). For purposes of control, measures should assess inputs that are regulated (e.g., budget authority or FTE). For purposes of allocation, measures should be based on efficiency (outputs or outcomes divided by inputs). Learning measures should uncover deviances from what is expected; motivational measures should compare real-time actual performance with targets; and promotion should illuminate aspects of performance that is most important to stakeholders.17
Using the wrong type of measure could lead to suboptimal decisions and lower performance. Returning to our example, Figure 8 mixes an evaluative
16
measure (readiness) with a control measure (spending rate), but not in a fashion that shows how the control affects the object being evaluated.
Lastly, it is very important that measures reflect desired outcomes. If the measures are not driving action that affects outcomes, then the system is not working effectively. To get to that point requires a mapping of the causal relationships between inputs and outputs, outputs and outcomes.
Recommendations
The SWE has established a large set of measures and has some experience working with them in different ways. There exists a significant experiential baseline. At this stage, the key users and generators of the information should shift attention to the usefulness and quality of the measures. Ask, “Do the data convey information that, given a decision-maker’s role and experience, enables him/her to make better decisions?” Just as we have described the chart above, conduct similar analyses of other displays of data to assess the quality of the information. Discard measures that are not useful or modify them so that they are.
Consider the actions a particular measure intends to drive and ask how best to characterize that measure to induce the proper action. The hierarchy of data-information-knowledge described earlier actually flows in both directions. One can collect and assemble data, put it in context to create information that, combined with experience, enables action. But, only by tapping one’s knowledge will one know whether information is useful and in turn what data should be collected. Thus, evaluation of the measures should be conducted in both directions. First, ask who would find a given measure useful for what types of decisions. Second, ask what types of decisions are being made and what data would best support them.
We recommend you map these relationships and document them as part of the basis for measurement. Maintaining process discipline is important. Once the causal relationships are mapped, establish a process to test them. If your mapping
suggests that when action X occurs, outcome Y results, then that should be
confirmed with experiments, tests or simulations. This is particularly important when action X is the addition or subtraction of resources.
It is a common belief that more money will buy more readiness. Building and testing such causal maps will reveal measures that are truly sensitive to changes in financial resources. It is very likely that some measures will not improve solely based on additional funding, whereas other measures may improve markedly. Studying the variances in spending patterns against variances in performance will help identify where the allocation financial resources has the greatest effect as oppose to the allocation of other resources (e.g., labor, equipment, management attention, or training). For example, the field of education has not definitively demonstrated that more money results in better education – that causal relationship is unproven – but other factors such as parental involvement or student:teacher ratios are effective. Figure 10 on page 28 is an example of such a mapping of causal relationships.
Just as the redline effort was worthwhile for understanding minimally acceptable levels of readiness, a “greenline” or “gold star” effort could help
determine the factors that yield excellence. Redline efforts help the mediocre unit avoid failure, but will not necessarily yield information to push them to greatness. At least as much attention should be paid studying the ships at the other end of the bell curve – the top performers – to gain insights that can help improve all the units.18
Finally, we recommend the enterprise consider how measures could be gamed or manipulated to ill effect and then establish countermeasures or controls to prevent that from occurring. To illustrate, we heard from a former commanding officer that mandatory jobs during an availability period had an MFOM value of 100 assigned to them. The ship scored a zero until the job was completed. While this
may effectively motivate the ship to perform that job, it does so by perverting an evaluation measure into a control measure. The value of MFOM as an accurate portrayal of the state of the ship is now lost. Rather than distort the meaning of MFOM, a different control device should be used for mandatory jobs. Such a device could be as simple as a checklist.
Additional Information
Analyzing Outcome Information: Getting The Most From Data by The Urban Institute
“Why Measure Performance?” by Bob Behn, appears in Public Administration Review.
Public Productivity Handbook by Mark Holzer.
Governance and Performance: New Perspectives by Carolyn J. Heinrich & Laurence E. Lynn, Jr.
Finding 4: Manage data quality, particularly cost data Description.
In many instances, scores are presented to the nearest tenth of a point on a 0-100 scale. One can see that in Figure 9. Is the SWE confident that they have the fidelity of data to track a ship to 1/1000th of a unit of measure? Conversations with members of the enterprise and limitations acknowledged by the enterprise suggest that the answer is clearly no.
We heard several reports and saw evidence of data quality issues. Some databases suffered from empty fields; maintenance jobs with multiple job control numbers had costs and efforts misallocated; or data did not accurately
cross-reference with related systems. When asked about maintenance cost estimating, we were told the standard for accuracy was +/- 40 percent. None of these issues is surprising and such problems plague most complex organizations with many legacy IT systems.
Looking specifically at financial resources, we noted that obligations were normally used as a proxy for cost. As noted earlier, costs are borne when resources are consumed in the activities of converting inputs to outputs. That is, cost is the consumption of inputs. An obligation, on the other hand, is the legal status of an appropriation and often occurs weeks or months before costs are actually borne, particularly in the areas of maintenance and supply where parts have a procurement lead time or sit in inventory. Services are obligated in full when contracts are let, but costs are borne more uniformly throughout the performance period. Other costs, including significantly large ones like sailors’ salaries, are not considered at all since the obligation is centrally managed by the Navy.
Consequence
The main consequence of data quality problems is the proverbial “garbage in, garbage out” phenomenon. Decision-quality can suffer if it is based on unreliable data. The field of medicine uses the term iatrogenics to describe the negative side effects of the primary treatment. (We have all heard the warnings in advertisements about sudden losses in blood pressure or feelings of dizziness.) Likewise, reliance on low quality or poorly selected data can have ill effects on the organization. One was described under the last finding: the distorted incentives in stoplight chart coding of data. Others include:
Tunnel vision: when managers, faced with many different targets, choose the ones that are easiest to measure and ignore the rest.
Sub-optimization: when managers choose to operate in ways that serve their own operation well but damage the performance of the overall system.
Myopia: when managers focus their efforts on short-term targets at the expense of longer-term objectives.
Measure fixation: when outcomes are difficult to measure there is a natural tendency to overly focus on the measurable outputs rather than
Misinterpretation: when the measures are imprecise there may be no real difference between the units compared, although this may not be obvious from the single-point estimates used
Gaming: can take many forms, from re-definition to deliberately setting low targets to misrepresentation.
Ossification: when an indicator has lost its purpose, but no one can be bothered to revise or remove it.19
Managing cost requires a cost accounting system. Cost accounting is different from budgetary (obligation-based) accounting. One cannot find cost
information in obligation-based accounting systems like STARS; one has to look for cost information in databases that track the use of inputs. Obligation-based
accounting systems generally only provide three pieces of information: budget authority available, obligations (promises to pay), and expenditures (actual payments). Such systems are designed to support the fiduciary responsibility of government financial managers, not the managerial roles for government general managers. In contrast, cost-based accounting systems contain information about processes or products, the direct inputs to those processes or products, the indirect inputs allocated to processes or products, and the amount paid for those inputs. Cost-based accounting systems tell you the cost of what you used and, thus, the cost of what you accomplished; obligation-based accounting systems only tell you the spending authority you had and how much of it you used, but it does not tell you what you own or what you did with it. It is critical to note that our recommendation for a cost accounting system is in addition to, rather than a replacement for, obligation accounting systems which remain necessary for fiduciary purposes.20
19
M. Pidd (2005) “Perversity in Public Service Performance Measurement” International Journal of Productivity and Performance Management 54(5/6):482-493.
20
Recommendations
Precision does not infer nor should it substitute for accuracy. Data should be rounded to represent the significance of the computation and the uncertainty in the data. Little that is done by the enterprise seems to warrant precision to the tenth of a percentage. In most cases rounding to the nearest whole number or a quantile of 20 or 10 is good enough for decision-making. Such rounding would reinforce to
decision-makers the level of certainty due to incomplete or unreliable data. The more uncertainty, the broader the quantiles should be. “Between 80 and 85” may be more accurate than 84.3.
Continue efforts to clean databases to ensure required database fields are complete and accurate. Implement monitoring and control systems (or strengthen the enforcement of existing ones) for database reliability. The Six Sigma portion of Lean Six Sigma is about reducing variability, which requires reliable and consistent efficiency measures.
Navy Enterprise as a whole should coordinate its efforts, and – more
importantly – communicate its information requirements, to those who are managing the accounting reforms in the Department of the Navy. The Navy’s Financial
Improvement Program, the DoN component of DoD’s Financial Improvement and Audit Readiness effort should be a partner with Navy Enterprise to ensure
managerially meaningful accounting data is generated by improved accounting systems. Navy ERP, a program of record that is deploying improved enterprise-wide accounting functionality should also partner with Navy Enterprise to ensure this multi-billion dollar investment provides the information most needed by readiness and provider enterprise leadership. Ad hoc efforts such as the common cost management framework should be abandoned and the efforts instead put toward long-term, systemic improvement.
Technical Adequacy – the measure is unique, complete, reliable and accurate; the measure accurately captures what you want to know about
Practicality – it is relatively easy and inexpensive to capture this measurement
User Independent – the data describe something that is comparable across entities (different ships) or across time (same ship in different time periods); it should be sensitive to changes (for budget purposes, adding or subtracting funding should cause this measure to move), and it is clearly understood by users and/or stakeholders.
User Dependent – the measure is relevant to the decision at hand or will drive a action when action is needed and it will do so in a timely fashion.21
Additional Information
“Performance Measures for Budget Justifications: Developing a Selection Strategy,” by Gloria Grizzle, appears in Public Productivity and Management Review
Cost Accounting by M. W. Maher & E. B. Deakin
How to Lie With Statistics by Darrell Huff and Irving Geis
Finding 5: Consider employing forward-looking and subjective indicators
Description.
One of the criticisms of the ships operations model used for budgeting is that it is backward-looking. A future naval force, consisting of different classes of ships, operated according to the fleet response plan is not well served by a model that describes resource requirements based on old operating models and different classes of ships. While it is a common practice to run a descriptive model in reverse to project a future trend, it is only valid if the envisioned future is bounded by the
21
Gloria Grizzle (1985) “Performance Measures for Budget Justifications: Developing a Selection Strategy,”Public Productivity and Management Review 9: 328-341.
range of data used to construct and validate the model. That is not the case with the ships operations model.
As it happens, the model is not a very accurate descriptive tool, let alone a predictive one. Several studies have identified deficiencies in the accuracy of the model.22 Until such time as a new model is constructed and goes through the
validation, verification, and accreditation process,23 the model should be augmented
with other indicators.
The use of such models appeals to the hyper-rational mindset common within the military. It is reinforced by thoughts that unless something can be measured, it cannot be managed and the corresponding belief that objective evidence is more accurate or valuable or useful than subjective evidence. Certainly, inputs can be measured objectively and outputs can be measured and counted. Outcomes,
however, are more elusive. As one digs deeper into the measurement of inputs and outputs, though, even things as seemingly concrete as cost become subjective as decisions must be made on how to allocate indirect costs. If the outcome the SWE aims for is warships ready for tasking, how does one allocate the cost of the Naval Surface Force staff, for instance?
Other governments have found it very useful to include subjective measures within the set of thing they consider. Subjective measures are most often used to assess outcomes rather than inputs, activities or outputs. An example from the city of Kirkland, Washington is provided in Figure 10.
22
Among them are: Hascall, A. M., Matthews, A. M., Gyarmati, M., Gantt, W. K., and Hajdu, Z.
Analysis of Ship Ops Model’s Accuracy in Predicting U.S. Naval Ship Operating Cost. MBA Professional Report, Naval Postgraduate School, June 2003. Kuker, K. L. and Craig D. H. A Feasibility Study of Relating Surface Ships OPTAR Patterns to their Operating Schedules. Master’s Thesis, Naval Postgraduate School, Monterey, California, June 1998. Williams, T. D. An Analysis of Selected Surface Ship OPTAR Obligations and their Dependency on Operating Schedules and Other Factors. Master’s Thesis, Naval Postgraduate School, Monterey, California, June 1987.
Figure 10: Example of Kirkland, Washington Performance Measures We see in this example that the focus of the measurement system is directed toward the ultimate outcome of citizens’ feeling of safety. The boxes at left describe what they believe to be one of the causal relationships affecting a feeling of safety, the role of the police department. The set of measures at right is hierarchically organized and includes input, output and outcome measures over time. One should note that not all measures were captured at all times – they have been evolving this system for several years. One should also notice that while their performance
