8
Performance measures are often seen as an administrative burden and additional cost to the 9
system investment. However, performance metrics, if reflective of the stakeholder requirements, 10
provides awareness of the value of the investment and enables effective management for 11
operations and maintenance. 12
“You cannot manage what you cannot measure.”
13
– Anonymous
14
The Enterprise Architecture and Geospatial communities of practice have several performance 15
indices that may be used in part or whole to help design and develop meaningful measures for 16
investments. Performance indices (e.g., Maturity Models) may provide a normalizing or level- 17
setting functional for an organization to better understand the range of capabilities and 18
investments and also contribute to the baseline assessment activities (see Section 3.4) when an 19
organization performs its Operational Requirements Document (Section 3.3). 20
One of the challenges of any maturity model is the general lack of a Return on Investment (ROI) 21
indicator for moving from one level to the next in the maturity progression ladder. A maturity or 22
capability model may have ~ 5 levels of maturity and while an organization assesses its maturity 23
to be a “3” there are generally no explicit cost/benefit to determine the value proposition for 24
moving to the next level. In fact, there may be diminishing returns and the stakeholders will need 25
to determine the optimal level of geospatial proficiency that meets the needs of the entire 26
investment. However; a performance management framework for geospatial capabilities that are 27
embedded within a larger system environment may include the necessity to tie the value to the 28
overall or ‘parent’ enterprise architecture investment. 29
9.3.1 OFFICE OF MANAGEMENT AND BUDGET:
1
ENTERPRISE ROADMAP
2
OMB’s Memorandum for Increased Shared Approached to Information Technology Services185
3
“provides Federal Agencies with policy guidance and management tools to use in increasing 4
shared approaches to information technology (IT) service delivery across mission, support, and 5
commodity areas.” The policy memo directs Federal Agency Chief Information Officers to submit 6
an “Enterprise Roadmap” each year that documents an agency’s current and future views of its 7
business and technology environment from an architecture perspective. In the 2013 submission 8
of the Enterprise Roadmap includes:186
9
1. Business and Technology Architecture (Main Body): a high-level, integrated description of 10
the agency’s IT-related strategic goals, business objectives, and enabling IT capabilities 11
across all operating units, and program areas. 12
2. Enterprise Architecture (EA) Maturity Measurement Template (Appendix 1): a self- 13
evaluation of the maturity of the Agency’s EA Program. 14
3. EA Outcomes and Measurements Template (Appendix 2): a self-evaluation of the 15
effectiveness of the agency’s enterprise architecture program, examples of contributions 16
to beneficial outcomes, areas for improvement, and measurement of value using the 17
attached template. 18
4. IT Asset Inventory (Appendix 3) (Optional): a list of IT systems and applications that 19
support mission, administrative, and commodity IT services, using the attached template 20
and the Federal Enterprise Architecture Reference Models that are provided in the 21
Common Approach. This Appendix will be considered “For Official Use Only.” 22
The EA Maturity Measurement Template (see Appendix G.1) provides a matrix that includes the 23
primary evaluation categories (e.g., Spending, Systems, Services, Security) and requires the 24
inclusion of IT investment Inventory and Outcomes with descriptions for; Area of Measurement, 25
Specific Measurement Indicators, Measurement Method and Targets (Timeline), and 26
Comments/Artifacts. Depending upon the category of Inventory or Outcome, the Areas of 27
Measurement may include: 28
Table 9-1. EA Maturity Measurement Template: Areas of Measurement
29
INVENTORY & OUTCOME AREA OF MEASUREMENT
Inventories Completeness
Accuracy Ratio
185 OMB Memo: Increasing Shared Approaches to Information Technology Services, May 2, 1012, available at
http://www.whitehouse.gov/sites/default/files/omb/assets/egov_docs/sharedapproachmemo_0502.pdf
INVENTORY & OUTCOME AREA OF MEASUREMENT
Outcomes Cost Savings/ Avoidance
Reduction of Duplication Efficiency
IT Enablement
9.3.2 ISE INFORMATION INTEROPERABILITY
1
FRAMEWORK: INTEROPERABILITY MATURITY
2
MODEL
3
The Information Sharing Environment (ISE) Interoperability Framework (I2F) is used to guide the
4
implementation of the ISE information sharing capabilities.187 The ISE I2F leverages existing
5
systems architecture guidance, suggested standards, tools, and methodologies to foster the 6
linkage of systems as well specifying the development of common artifacts that are intended to 7
enable disparate architectures to improve information sharing. 8
The Interoperability Maturity Model of the ISE I2F is aligned with the OMB guidance Federal
9
Enterprise Architecture Framework and The Common Approach to Federal Enterprise Architecture
10
and is broken down by domains (e.g., Business, Data, Applications, Infrastructure, Security, and 11
Performance). The model establishes characteristics for each level of interoperability (e.g., ad hoc, 12
repeatable, enhanced, managed, and optimized) for each interoperability requirement. Each row 13
in the maturity model represents a functional area within the domain. Each column represents a 14
different stage of maturity. Interdependences between functional areas exist but the goal is to 15
assess a system independently for each functional area. 16
Within the I2F Performance Domain maturity model, it is divided into functions or process groups
17
(rows) and maturity levels (column). The maturity model is then followed by several supporting 18
questions. 19
Table 9-2. I2F Performance Domain Maturity Model Metrics
20 ⓪ ABSENT ① AD HOC ② REPEATABLE ③ ENHANCED ④ MANAGED ⑤ OPTIMIZED Metrics Formalized performance metrics that provide direct line of sight between strategic planning and the investment review process do not exist.
Formalized
performance metrics exist and align with strategic goals of organization as well as to applicable policy, guidance, and laws.
Formalized performance metrics that identify common performance elements across investments or activities exists.
Formalized performance metrics are used to inform gap analysis of interoperability
requirements and adhere to relevant performance goals.
187 Program Manager – Information Sharing Environment (PM-ISE), Information Interoperability Framework (I2F), Version 0.5, March
9.3.3 DHS COMMON OPERATING PICTURE: KEY
1
PERFORMANCE INDICATORS AND MEASURES
2
The Department of Homeland Security has chartered an Executive Steering Committee (ESC) for 3
its Common Operating Picture (COP)/User Defined Operating Picture (UDOP) Domain. The 4
Components of DHS have invested in multiple COP/UDOP capabilities to support situational 5
awareness for law enforcement, emergency management, intelligence, and homeland 6
security/defense mission activities. The goal of the COP ESC has been to increase COP 7
interoperability, effectiveness, and shared capabilities while reducing the Department’s collective 8
operational costs by managing COP systems as enterprise mission service investments. It is 9
expected to promote and guide the development and operation of, and investment in the DHS 10
Common Operating Picture domain. The COP ESC will provide analytical support and provide 11
recommendations, guidance, and procedures for improving the sharing of data, information, 12
infrastructure, tools, and services across DHS COP investments. 13
In support of the COP ESC, the DHS Geospatial Management Office prepared the DHS Sensitive
14
But Unclassified COP/UDOP Segment Architecture188 document is to provide a holistic and
15
conceptual view of the future consolidated or interoperable state of the COP domain for 16
Homeland Security. This initial version focused upon the target technical architectural areas: 17
business, data, services, technology, security, and performance. It presents a Target Architecture 18
based on a common services framework that relies on shared services and enterprise delivery of 19
core data, software, and infrastructure using approved standards. This shared services approach 20
ensures data and system interoperability and reliable exchange of information in a usable and 21
geospatial format. 22
The target performance architecture for the DHS COP segment architecture prepared a 23
performance management scorecard (Appendix G.2) to tracking progress and effectiveness 24
toward achieving the strategic goals and objectives for the COP domain. The scorecard is based on 25
the key performance indicators (KPIs) to include; Governance, Information Sharing, Mission 26
Enablement and Technology Management. These scorecard metrics were established from the 27
COP Domain priorities for interoperability, effectiveness, authoritative/trusted information in a 28
geospatial format, standards-based information exchanges, reliability, and shared capability. 29
9.3.4 NATIONAL STATES GEOGRAPHIC INFORMATION
30
COUNCIL: GEOSPATIAL MATURITY ASSESSMENT
31
The National States Geographic Information Council’s (NSGIC) Mission is to promote statewide 32
geospatial coordination activities in all states and to be an effective advocate for states in national 33
geospatial policy and initiatives, thereby enabling the National Spatial Data Infrastructure 34
188 Department of Homeland Security, COP/UDOP Sensitive But Unclassified Segment Architecture, Version 1.0 (DRAFT), April 27, 2012,
(NSDI).189 NSGIC maintains a Geospatial Maturity Assessment (GMA) that is a “baseline
1
assessment methodology to routinely and continuously monitor and validate statewide geospatial 2
capabilities.”190 The GMA included eighty three (83) questions that characterized their geospatial
3
programs. The assessment is over one-half data focused, but also includes questions on staffing 4
and budget, strategic and business planning, and interagency coordination and data sharing. 5
9.3.5 URBAN AND REGIONAL INFORMATION SYSTEMS
6
ASSOCIATION: GIS CAPABILITY MATURITY MODEL
7
The Urban and Regional Information Systems Association (URISA)191 promotes the effective and
8
ethical use of spatial information and information technologies for the understanding and 9
management of urban and regional systems. URISA members and participants typically use 10
geospatial and other information technologies to solve challenges in government agencies. URISA 11
provides educational programs, offers volunteer GIS expertise through its GISCorps program, and 12
assists government agencies with benchmarking GIS maturity through its GIS Management 13
Institute®. 14
“The URISA GIS Capability Maturity Model192 is a key component of the GIS Management
15
Institute. Its primary purpose is to provide a theoretical model of a capable and mature enterprise 16
GIS operation within a designated organization. The URISA Model is intended to serve the GIS 17
community as a stand-alone document to define the framework for an effective enterprise GIS. 18
The Model was developed initially with a focus on local government agencies (e.g., cities, 19
counties, regional agencies, and other similar entities) but it is intended for future use by any 20
enterprise GIS. As a stand-alone document, the Model is intended to facilitate discussion amongst 21
GIS managers and the decision makers who deploy and fund GIS to maximize effectiveness and 22
return on investment from a given level of investment.” 23
The Capability Maturity Model assumes two (2) broad areas of GIS operational development: 24
• Enabling Capability – the technology, data, resources, and related infrastructure 25
that can be bought, developed, or otherwise acquired to support typical enterprise 26
GIS operations. Enabling capability includes GIS management and professional staff. 27
• Execution Ability – the ability of staff to utilize the technology at their disposal 28
(subject to separate assessment as part of the Model). 29 189 http://www.nsgic.org/ 190 http://www.nsgic.org/geospatial-maturity-assessment/ 191 http://www.urisa.org/main/about-us/ 192 http://www.urisa.org/clientuploads/directory/GMI/GISCMM-Final201309%28Endorsed%20for%20Publication%29.pdf