Application-oriented Evaluation Using Goal-Question-Metric Approach

Because the governance reference model was built based on a set of strategies developed from stakeholder input, it is important that the implementation of the strategies be verified. Each strategy was documented in the Analysis phase, and each will be verified by documenting its impact on the prototype. The original strategies table will be expanded as follows:

Table 10: Impact Documentation Template <No.>

Name: <Issue name>

Description: <Description of issue> Influencing

Factor(s):

<The factor or list of factors that affect this issue> Solution: <Discussion of a general solution to the design issues,

followed by a list of the associated strategies> Strategy: <Explanation of the strategy>

Related Strategies: <References to related strategies and a discussion of how they are related to this issue>

Impact: <Explanation of how the implementation of this strategy affected the implementation of the prototype>

The impact will be determined using an application-oriented evaluation based on part of the reference model supply chain methodology developed by Bohmann, Schermann, and Kremar (Bohmann, 2007). The methodology incorporates the Goal- Question-Metric (GQM) approach.

GQM is an approach to software metrics that has been promoted by Victor Basili of the University of Maryland, College Park and the Software Engineering Laboratory at the NASA Goddard Space Flight Center (Basili, 2002).

GQM defines a measurement model on three levels: 1. Conceptual level (goal)

A goal is defined for an object for a variety of reasons, with respect to various models of quality, from various points of view and relative to a particular environment.

A set of questions is used to define models of the object of study and then focuses on that object to characterize the assessment or achievement of a specific goal. 3. Quantitative level (metric)

A set of metrics, based on the models, is associated with every question in order to answer it in a measurable way.

The open literature typically describes GQM in terms of a six-step process where the first three steps are about using business goals to drive the identification of the right metrics and the last three steps are about gathering the measurement data and making effective use of the measurement results to drive decision making and improvements. Basili described his six-step GQM process as follows:

1. Develop a set of corporate, division and project business goals and associated measurement goals for productivity and quality

2. Generate questions (based on models) that define those goals as completely as possible in a quantifiable way

3. Specify the measures needed to be collected to answer those questions and track process and product conformance to the goals

4. Develop mechanisms for data collection

5. Collect, validate and analyze the data in real time to provide feedback to projects for corrective action

6. Analyze the data in a post mortem fashion to assess conformance to the goals and to make recommendations for future improvements

GQM templates are a structured way of specifying goals. A GQM template contains the following fields:

Table 11: Goal-Question-Metric Template

Field Examples

object of study pair programming, static analysis tool purpose characterize, understand, evaluate, predict,

improve

focus programmer effort, program reliability

stakeholder developer, customer, manager

context factors other important factors that may affect outcomes

For the evaluation of the governance reference model, this dissertation will focus on the second stage in the reference model supply chain – solution design. In the solution design, the outputs of applying the reference model are of interest. According to Misic and Zhao, development and application of reference models is motivated by the prospect of reducing cost, enhancing revenues, or minimizing risks (Misic, 2000). In order to evaluate the output of reference modeling, the goal of this stage is improving cost, time, and Quality of Service (QoS) of the data service, which has to be supported by applying the governance reference model (DoD CIO, 2006; Choi, 2008; Misic, 2000; Menasce, 2002; Menasce, 2007). The data services created using the prototype will be evaluated based on the following GQM matrix (Bohmann, 2007; Basilli, 2002; Choi, 2008):

Table 12: Data Service Goal-Question-Metric Matrix (Menasce, 2007; Choi, 2008)

Goal Purpose Improve

Issue time, cost, and quality of

Object / Process data service by using governance reference model Viewpoint from a designer‘s (service developer) point of view Question 1 Did the application of the governance reference model affect the

development time of the data service?

Metric 1 Time needed to discover required standards and policies (TS)

Metric 2 Time needed to identify required run-time metrics (TM)

Metric 3 Time needed to assign roles and responsibilities (TR)

Question 2 Did the application of the governance reference model affect the development cost of the data service?

Metric 4 Approximate cost associated with the data service development ((TS + TM + TR) x average hourly rate))

Question 3 Did the application of the governance reference model affect the QoS of the data service?

Metric 5 Availability:

o Data Service Request Count

Metric 6 Performance:

o Data Service Average Response Time

Metric 7 Reliability:

o Data Service Response Count / Data Service Failure Count

The metrics data for Time (TS, TM, and TR) will be collected by the prototype using a time-watch function that will start when a new service is created, and will stop when the new service is submitted into the registry. The metrics data for Cost is determined by calculating the product of the total Time metrics and average hourly rate (based on industry standard). The metrics data for Availability, Performance, and Reliability will be collected by the prototype application based on standard computing functions (Bohmann, 2007; Basilli, 2002; Choi, 2008).