4 Service Transition
4.2 Evaluation
4.2.1 Process Overview
The purpose of the Evaluation process is to provide a consistent way of determining the performance of a service change pertaining to current and upcoming services and supporting infrastructure.
It is a generic process that considers whether the performance of something – a service or component – is acceptable, fit for purpose, financially viable. It determines the performance of a service change and evaluates service quality on a regular basis. This includes identifying areas where the targeted service levels are not reached, and holding regular talks with business to make sure the agreed service levels are still in line with the business needs.
This process delivers an important piece of input for Continual Service Improvement and the future improvement of Service development and Change Management. It includes three parts:
• Complaints Management; to assess customer complaints and to instigate corrective action if required
• Customer Satisfaction Survey; to plan, carry out and evaluate regular customer satisfaction surveys. The principal aim of this process is to learn about areas where customer
expectations are not being met before customers are lost to alternative service
• Service Review; to review business services and infrastructure services on a regular basis.
The aim of this process is to improve service quality where necessary, and to identify more economical ways of providing a service where possible.
4.2.2 High Level Process Model
The following diagram illustrates the high level process model.
7. Implement Corrective Actions
1. Define what you should measure.
6. Present and Use the information.
2. Define what you can measure.
ITIL 7 Step Process
5. Analyze the data. 3. Gather the data.
4. Process the data.
Figure 13 Evaluation Model
4.2.3 High Level Process Descriptions
The sub-processes of the high level process model are described at a high level.
Number Sub-process Description
1.0 Initiate Evaluation Initiate the Evaluation Process from one or more of the following processes: Change Management, Service Design or Service Validation & Testing.
2.0 Request for Change (artifact) Receive a RFC from Change Management for a change to an existing service or the introduction of a new service.
3.0 Service Design Package (artifact)
Receive a Service Design Package (SDP), which provides the acceptance criteria and predicted performance to be evaluated, from Service Design.
4.0 Test Plan and Results (artifact) Receive the test plans and results related to the new or modified service to be evaluated from Service Validation & Testing.
5.0 Plan the Evaluation Develop an evaluation plan appropriate for the type of evaluation. Develop a recommended list of test beds at which to perform the test.
6.0 Evaluate Predicted Performance
Evaluate the predicted performance expected from the evaluation and if the activity determines that the results are not acceptable, the evaluation is
terminated and Change Management is notified of the results.
7.0 Predicted Performance OK? Determine whether the predicted performance meets established criteria and forward findings to Change Management. Obtain decision from Change Management whether to proceed.
8.0 Interim Evaluation Report Generate an interim evaluation report of the actual performance evaluations and acceptable results that includes recommendations. Change Management is notified of the results and is provided the final report.
9.0 Evaluate Actual Performance Evaluate the Actual performance expected from the evaluation and if the activity determines that the results are not acceptable, the evaluation is
terminated and Change Management is notified of the results.
10.0 Actual Performance OK? Determine whether the predicted performance meets established criteria and forward findings to Change Management. Obtain decision from Change Management whether to proceed.
11.0 Interim Evaluation Report (artifact)
An interim evaluation report is generated of the actual performance evaluations and acceptable results that includes recommendations
12.0 Evaluation Report (artifact) The final evaluation report is generated of the actual performance evaluations and acceptable results that include recommendations.
Communicate results of final report to change management.
4.2.4 Seam Management Authority Matrix
The term seam refers to a connection or interaction between one or more organizations. It could be an interaction between Government (DON) organizations, Government (DON) to IT Service Provider, or IT Service Provider to IT Service Provider. The seam analysis helps to identify the processes, people and technology required to coordinate end to end management of IT Services.
The following table provides key seam descriptions.
Seam Seam Descriptions
TBD
The following tables identify key process activities and where possible, associate activity with process seams and key roles.
Seam Required ITSM Process Specification
The Service Provider segment Change Manager shall be the single point of contact for each Release. This is the operational role that coordinates activities and
supervises resources in the performance of Process Activities within their segment.
The authority matrix below identifies process activities that can be associated with key roles and/or process seams. These relationships are documented in terms of RACI (Responsible, Accountable, Consulted, and Informed). This table does NOT represent a RACI mapping to the organization or sub-processes.
Seam Required ITSM Process Process Process Service Provider
Change
Specification Owner Manager Lead Manager Assist the DON with identifying the
business, financial and technical criteria needed to evaluate the Service Provider’s new or changed Service.
CI AR R R
Assist the DON with the planning, evaluation and reporting for all new or changed Services involving the Service Provider. This may include setting up testing environments, loading test data, and system configuration.
CI AR R R
Ensure all information related to the new or changed Service is up-to-date within the SKMS.
CI AR R R
Support the DON in evaluating deviations between predicted and actual performance for all new or changed Services.
CI AR R R
Provide Evaluation and Interim Evaluation reports at the request of the DON in a DON-defined format that contains the following sections:
• Risk profile (A representation of the residual risk left after a Change has been implemented and after countermeasures have been applied)
• Detailed and Summary analysis of Deviations (The difference between predicted and actual performance following the implementation of a change)
• Qualification statement (proof that a change to a service does not adversely affect the quality or standards of that service or related services of other providers)
• Validation statement (statement that the Service can be validated based on validation test results and a validation plan)
• Recommendation (statement recommending accepting or rejecting the new or changed Service)
CI AR R R
4.2.5 Process Interfaces
Any process is triggered by an event or activity and associated artifact(s), and produces an output which typically feeds into another process.
Interfaces
This process has a key relationship or dependency with the following ITSM processes:
• Change Management
• Service Design
• Service Validation and Testing Inputs
Consider the inputs to the process:
• Service Package
• Release package
• Service Design Package (SDP)
• Request for Change (RFC)
• Service Acceptance Criteria
• Test Plan and Results Outputs
Consider the outputs from the process:
• Evaluation report
4.2.6 Roles
The following roles have been identified with the process.
Roles Description
Evaluation Process Owner The strategic role accountable for the process.
Evaluation Process Manager The tactical role that performs cross segment coordination.
Change Manager A Change Manager has the responsibility for management reports, chairing all CAB meeting, tabling RFCs,
receiving, logging and allocating priority, authorizing or rejecting RFCs, establishing schedules, tracking progress and reviewing.
4.2.7 Performance Metrics
The effectiveness and performance of the process are measured using metrics-based Key Performance Indicators (KPIs) which support high level Critical Success Factors (CSFs). The CSFs and KPIs below are examples which may or may not reflect actual Key Performance Parameters (KPPs).
CSF # Critical Success Factors KPI # Key Performance Indicators
1 Improve Customer Satisfaction 1 Number of Customer Complaints - Number of received customer complaints.
2 Number of Accepted Customer Complaints - Number of received customer complaints which were accepted as justified.
3 Number of Customer Satisfaction Surveys - Number of formal Customer Satisfaction Surveys carried out during the reporting period.
4 Percentage of Returned Questionnaires - Percentage of questionnaires returned, in relation to all questionnaires being sent out.
5 Number of Service Evaluations - Number of formal Service Evaluations carried out during the reporting period.
2 Improve accuracy in responding to Customer needs
6 Number of Process Benchmarkings, Maturity Assessments, and Audits - Number of formal Process Benchmarkings, Maturity
Assessments, and Audits carried out during the reporting period.
7 Number of Process Evaluations - Number of formal Service Evaluations carried out.
8 Number of Identified Weaknesses - Number of weaknesses which were identified during Service Evaluation, to be addressed by improvement initiatives.
3 Increase response times to Customer 9 Variance from service performance required by customers
10 Number of Incidents against the service 11 Number of failed designs that have been
transitioned
12 Cycle time to perform an evaluation
4.2.8 Tools Interface Requirements
Tool interfaces requirements cannot be specified in any detail as Evaluation can be such a broad area.